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FOREWORD 


Among  the  responsibilities  assigned  to  the  Office  of  the  Manager,  National 
Communications  System,  is  the  management  of  the  Federal  Telecommunication 
Standards  Program.  Under  this  program,  the  NCS,  with  the  assistance  of  the 
Federal  Telecommunication  Standards  Committee  identifies,  develops,  and 
coordinates  proposed  Federal  Standards  which  either  contribute  to  the 
interoperability  of  functionally  similar  Federal  telecommunication  systems  or  to  the 
achievement  of  a  compatible  and  efficient  interface  between  computer  and 
telecommunication  systems.  In  developing  and  coordinating  these  standards,  a 
considerable  amount  of  effort  is  expended  in  initiating  and  pursuing  joint  standards 
development  efforts  with  appropriate  technical  committees  of  the  International 
Organization  for  Standardization,  and  the  International  Telecommunication  Union  - 
Telecommunication  Standardization  Sector.  This  Technical  Information  Bulletin 
presents  an  overview  of  an  effort  which  is  contributing  to  the  development  of 
compatible  Federal,  national,  and  international  standards  in  the  area  of  video 
teleconferencing.  It  has  been  prepared  to  inform  interested  Federal  activities  of  the 
progress  of  these  efforts.  Any  comments,  inputs  or  statements  of  requirements 
which  could  assist  in  the  advancement  of  this  work  are  welcome  and  should  be 
addressed  to: 

Office  of  the  Manager 
National  Communications  System 
Attn:  NT 

701  S.  Court  House  Road 
Arlington,  VA  22204-2198 
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1.0  INTRODUCTION 


1.1  General 

This  document  summarizes  work  performed  by  Delta  Information  Systems, 
Inc.  (DIS)  for  the  National  Communication  Systems  (NCS),  Office  of  Technology 
and  Standards.  The  NCS  is  responsible  for  the  management  of  the  Federal 
Telecommunications  Standards  Program,  which  develops  telecommunications 
standards,  whose  use  is  mandatory  for  all  Federal  departments  and  agencies. 

This  document  is  a  final  report  for  a  Task  Order  on  Contract  DCAl 00-91 -C- 
0031 .  The  title  for  the  contract  and  Task  Order  are  listed  below, 

■  Contract  DCAl 00-91 -C-0031 

Development  of  Federal  Telecommunications  Standard  Relating  to 
Digital  Facsimile  and  Video  Teleconferencing 

■  Task  No.  3 

Technical  Work  in  the  Area  of  Video  Teleconferencing 

■  Subtask  No.  3 
Performance  Testing 

1.2  Background 

In  recent  years,  with  the  onset  of  digital  television  processing  and 
transmission,  developments  in  television  practices  have  rendered  former  methods 
of  quality  measurements  of  television  signals  as  less  accurate  or  applicable,  as  one 
would  expect.  In  the  past,  quality  measurements  of  analog  television  signals  were 
easily  verified  by  use  of  video  test  signals  that  show  amplitude  and  transient 
response,  and  for  distortion  freedom  by  differential  gain  and  phase.  In  digital 
television  systems,  the  advent  of  bandwidth  compression  has  introduced 
techniques  such  as  interframe  coding  which  has  created  the  need  to  identify 
objective  measurements  that  can  be  used  to  accurately  verify  the  quality  of  the 
video  conducted  by  the  particular  digital  system  in  use. 

This  effort  is  undertaken  with  funding  from  NCS  as  part  of  a  continuing 
effort  to  establish  standards  for  T1A1  for  video  information  transmission.  The 
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results  of  these  efforts  will  also  be  available  for  general  scientific  use. 

The  purpose  of  this  subtask  is  to  investigate  the  results  of  subjective  tests 
with  objective  tests  made  under  the  same  conditions  of  a  selection  of  video  scenes 
that  are  considered  typical  of  video  telephony,  video  teleconferencing  or 
entertainment  video.  The  subjective  tests  are  reviewed  in  detail  in  Section  2.0  of 
this  report.  The  objective  tests  are  reported  in  detail  in  Section  3.0  of  this  report. 

DIS  has  developed  a  series  of  computer  generated  test  patterns  for  objective 
testing.  One  objective  test,  the  Scene  Cut  Response  (SCR)  is  used  exclusively  in 
this  report. 
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2.0  SUBJECTIVE  TESTING 


2.1  Executive  Summary 

The  Subjective  Testing  reported  herein  is  required  by  T1A1.5  to  support 
development  of  transmission  standards  for  Video  Telephony  and  for  Video 
Teleconferencing. 

The  method  for  accomplishing  scene  quality  grading  was  by  use  of  viewer 
reactions  to  numerous  conditions  affecting  video  coding  quality.  Three  LABs 
collected  viewer  opinion  of  scene  degradation  with  five  levels  of  scoring  the 
difference  between  an  original  scene  and  the  same  scene  processed  by  a 
Hypothetical  Reference  Circuit  (HRC),  as  follows:  5  -  Imperceptible;  4  -  Perceptible 
but  not  annoying;  3  -  Slightly  annoying;  2  -  Annoying;  or  1  -  Very  annoying. 

Viewer  impressions  were  screened  for  scoring  accuracy  through  use  of  no 
degradation  (nulls)  and  repeats.  Those  validated  scores  were  summarized  for 
statistical  character  and  compared  with  two  other  Test  LABs'  results. 

2.2  Test  Location 

The  Subjective  Testing  portion  of  Performance  Testing  rendered  by  Delta 
Information  Systems  (DIS)  was  one  of  three  sites  collecting  viewer  reaction  to  test 
scenes.  These  three  Test  Sites  are  identified  as  LAB  X  (Delta  Information 
Systems/National  Communications  System),  LAB  Y  (GTE),  and  LAB  Z  (Department 
of  Commerce/National  Telecommunications  &  Information  Administration/Institute 
for  Telecommunications  Sciences). 

Since  LAB  X  was  to  be  supplied  with  viewers  from  the  Washington,  D.C., 
area,  the  selection  of  the  test  location  was  necessarily  in  the  Washington,  D.C., 
vicinity.  Many  possible  sites  were  poor  seconds  to  the  "Expert  Viewing  Room"  at 
Advanced  Television  Testing  Center,  1330  Braddock  Place,  Alexandria,  Virginia. 
That  facility  maintained  viewing  conditions  in  accordance  with  CCIR 
Recommendation  500-5  and  was  outfitted  with  much  desired  secondary  support 
equipment  as  well  as  a  solid  technical  staff.  Also,  other  candidate  test  locations 
would  have  required  considerable  effort  to  comply  with  the  CCIR  Recommendation 
500-5. 
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2.3  Test  Video  Tape  Details 


The  source  of  video  used  for  Subjective  Testing  was  the  result  of  a  set  of 
still  images  from  25  scenes,  as  shown  in  Appendix  A.  These  25  scenes  represent 
five  categories  that  are  typical  of  video  telephony,  video  teleconferencing,  and 
entertainment  video.  These  categories  cover  a  wide  range  of  movement  and 
detail.  Each  scene  is  edited  to  a  length  of  nine  seconds.  These  categories  are: 

a.  one  person,  mainly  head  and  shoulders  (scenes  f,  ],  k,  I); 

b.  one  person  with  graphics  and/or  more  detail  (scenes  a,  s,  e,  m,  n,  w); 

c.  more  than  one  person  (scenes  d,  g,  o,  p,  q,  r); 

d.  graphics  with  pointing  (scenes  c,  s,  t,  u,  v,  w); 

e.  high  object  and/or  camera  motion  (scenes  h,  i,  y). 

Also  included  on  the  scene  tapes  are  candidate  objective  test  patterns  of  the 
rotating  wheel  and  three-size  dot  patterns.  Each  set  of  scenes  is  processed  by 
HRCs  chosen  to  span  a  broad  range  of  video  quality.  The  25  HRCs  represented 
the  following  categories.  These  systems  cover  a  range  of  bit  rates  from  112  Kb/s 
to  70  Mb/s.  These  categories  are: 


GROUP  NO.  CIRCUIT  DESCRIPTION  HRC  NO. 

1  High  quality  1-3 

2  Vector  quantization,  medium  rate  4,  5 

3  Proprietary,  low  to  medium  rate  6,  7 

4  Proprietary,  medium  to  high  rate  8-10 

5  QCIF,  low  rate  11-13 

6  QCIF,  medium  rate  14 

7  CIF,  low  rate  15-18 

8  CIF,  medium  rate  19-21 

9  CIF,  high  rate  22  -  25 

All  were  recorded  on  Beta  Cam  D2  format. 
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From  these  video  recordings,  three  sets  of  four  tape  recordings  were 
generated,  following  the  explanation  in  the  Subjective  Test  Plan,  Appendix  B 
(T1A1 .5/94-1 18R1  attached),  to  provide  a  balanced  set  of  video  scenes  for 
subjective  test  viewing.  These  tapes  are  identified  as  for  viewing  by  the  Red, 
Green,  or  Orange  Team.  The  scene  content  sequence  of  each  tape  is  shown  in  the 
Subjective  Test  Plan.  It  is  noted  that  a  viewer  will  see  scenes  as  processed  by  a 
restricted  number  of  HRCs.  It  is  also  noted  that  all  viewers  will  see  scenes 
processed  by  HRC  20  and  that  HRCs  4,  15,  and  17  are  viewed  by  two  of  the  three 
teams. 

2.4  Viewer  Scheduling 

Viewers  recruited  for  the  subjective  testing  conducted  by  DIS  were 
organized  by  the  management  of  NCS  from  Federal  Departments  in  the  vicinity  of 
Washington,  D.C.  The  initial  call  for  volunteers  for  the  subjective  test  yielded 
approximately  one-half  of  the  desired  number  of  participants.  With  the  test  studio 
schedule  pressing  a  start,  the  recruiting  effort  was  continued  with  use  of  the 
Equivalent  Method  of  Viewer  Randomization  put  into  use.  Those  who  expressed 
interest  in  the  program  supplied  an  availability  forecast  which  was  the  basis  for 
LAB  X  Team  assignments. 

Viewers  were  usually  scheduled  for  two  tape  plays  in  a  two-hour  period. 
Occasionally,  just  one  tape  was  viewed.  It  was  virtually  impossible  to  maintain  a 
team  schedule  due  to  the  viewer  availability.  A  practice  tape  was  presented  prior 
to  viewing  the  scheduled  tapes.  Viewers  were  seated  in  the  "Expert  Viewing 
Room,"  as  shown  in  Figure  2.1.  Data  sheets  for  each  tape  play  were  annotated  by 
the  viewer  for  their  opinion  of  each  presented  scene.  A  fifteen-minute  break  was 
included  between  tapes  for  those  viewing  two  consecutive  plays.  (See  Test  Plan, 
Appendix  B.) 

It  was  necessary  to  revise  the  viewing  schedule  daily  for  nearly  the  entire 
test  period.  A  total  of  46  individuals  were  scheduled  for  viewing.  Thirty-eight  of 
these  actually  saw  tapes.  Three  of  these  saw  only  2  of  the  4  tapes.  Only  five 
viewers  failed  the  null  check,  resulting  in  30  qualified  viewer  completions  -  10  for 
each  Team.  Demographic  data  for  the  35  viewers  scoring  4  tapes  is  shown  in 
Table  2-1 .  See  Table  2-2  for  the  final  completed  viewing  schedule. 


2  -  3 


RF  Obfribution  Panab 


Expert  Viewing  Room 


TABLE  2-1  DEMOGRAPHIC  FILE 


ID 

DECADE 

GENDER 

INDUSTRY 

JOB 

PAST 

VIDEO 

EYE 

TEST 

SESSION 

SEAT  ORDER 

Olx 

40 

female 

Government 

Technical 

Recent 

7/2 

center 

2143 

09x 

30 

female 

Broadcasting 

Professional 

No 

7/3 

right 

1234 

llx 

40 

male 

Newspapers 

Government 

Professional 

Recent 

7/4 

left 

1234 

26x 

30 

male 

Government 

Professional 

No 

8/2 

right 

2431 

33x 

40 

female 

Government 

Professional 

No 

7/4 

left 

2134 

24x 

30 

male 

Government 

Technical 

No 

7/2 

right 

1243 

29x 

40 

male 

Government 

Professional 

No 

8/3 

center 

1342 

3  Ox 

30 

male 

Government 

Professional 

Yes 

9/0 

center 

1423 

31x 

30 

male 

Government 

Technical 

No 

9/1 

left 

3412 

35x 

20 

male 

Government 

Professional 

Recent 

7/0 

right 

3412 

40x 

50 

female 

Government 

Secretarial 

No 

7/4 

left 

1234 

17x 

50 

male 

Government 

Administrative/ 

No 

6/1 

center 

1342 

41x 

20 

male 

White  Collar 
Telecommunication/  Professional 

No 

7/0 

right 

3412 

ID 

DECADE 

GENDER 

Utilities 

INDUSTRY 

GREEN  TEAM 

JOB 

PAST 

VIDEO 

EYE 

TEST 

SESSION 

SEAT  ORDER 

08x 

30 

female 

Government 

Administrative/ 

Recent 

7/3 

left 

1423 

23x 

30 

male 

Government 

White  Collar 
Executive/ 

No 

7/1 

right 

1432 

14x 

60 

male 

Government 

Managerial 

Executive/ 

Yes 

10/1 

center 

1432 

lOx 

60 

male 

Government 

Managerial 

Professional 

Yes 

8/3 

right 

2134 

27x 

40 

male 

Government 

Technical 

Recent 

7/1 

left 

2134 

04x 

30 

male 

Telecommunication/  Owner/ 

No 

6/3 

left 

1432 

12x 

30 

female 

Utilities 

Government 

Operator 

Secretarial 

No 

7/1 

center 

3214 

25x 

20 

male 

Government 

Professional 

No 

10/1 

center 

3214 

34x 

50 

male 

Electronics/ 

Professional 

Recent 

7/4 

left 

3214 
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Computers 


03x 

30 

male 

Government 

Technical 

Recent 

9/1 

center 

1432 

ORANGE  TEAM 

PAST 

EYE 

SESSION 

ID 

DECADE 

GENDER 

INDUSTRY 

JOB 

VIDEO 

TEST 

SEAT 

ORDER 

02x 

60 

male 

Government 

Professional 

No 

6/4 

right 

3421 

06x 

50 

male 

Government 

Executive/ 

Managerial 

Recent 

6/4 

right 

1423 

07x 

50 

female 

Government 

Administrative/ 
White  Collar 

No 

7/4 

left 

4321 

15x 

20 

male 

Government 

Professional 

Recent 

8/0 

right 

2314 

20x 

30 

female 

Government 

Clerical/ 

Support 

No 

10/1 

right 

4321 

2  lx 

60 

male 

Government 

Technical 

No 

7/1 

center 

4321 

22x 

20 

female 

Government 

Professional 

No 

8/0 

left 

3214 

28x 

40 

male 

Government 

Administrative/ 
White  Collar 

Yes 

7/4 

center 

4231 

37x 

20 

male 

Government 

Professional 

No 

10/0 

left 

4213 

38x 

20 

female 

Government 

Secretarial 

No 

8/1 

center 

4231 

32x 

50 

male 

Government 

Executive/ 

Managerial 

No 

7/4 

center 

1423 

36x 

20 

male 

Government 

Technical 

No 

6/3 

center 

4231 

2.5  Test  Procedure 
2.5.1  Equipment 

After  the  dispatch  of  required  equipment  and  other  material  to  the  Advanced 
Television  Test  Center,  the  test  video  system  vy/as  assembled  and  calibrated  for  the 
test  condition  requirements  of  CCIR  Recommendation  500-5  and  the  Subjective 
Test  Plan.  The  video  equipment  remained  powered  for  the  entire  duration  of 
subjective  testing  to  avoid  the  level  drifting  that  was  noted  for  the  first  two  hours 
following  the  initial  cold  start. 
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TABLE  2-2 
FINAL  SCHEDULE 


SUBJECT  I VE  SCHEDULE 


MAY  26 


DAY  HOUR  TAPE  L 


C  *  R 


MON  25  9 

1 0 
1  1 
12 

13 

14 

15 

1  ••i) 
17 


03 

04 

G3 

G2 


DIXDN 

BRACKNEY 

SAVOYE 

DIXON 

BRACKNEY 

SAVOYE 

LOPEZ 

POOLE 

LOPEZ 

POOLE 

T  UEl 


14 

15 

16 


b’l 

04 

R1 

R3 

R1 

04 

03 


lope  2 

CHAMP I ON 

LOPEZ 

CHAMP  I  ON 
BARR 

BARR 

JAMIL 

ORNDORFF 

MCCLELLEN 

LABONT 

UF<ND0F;F  F 

MCCLELLEN 

LhBUN }  i 

V 

Gi 

TART 

SEL I3KAR 

1  C) 

G4 

FART 

UEE  1  5KA:  < 

j.  1 

01 

OLDElN 

RYAN 

12 

04 

OLDEN 

RYAN 

1  •  *' 

R4 

BARR 

JAMIL 

i  4 

R2 

BARR 

JAMIL 

1  5 

G3 

TART 

SdEL  1  yK.  AF'! 

16 

1  / 

G2 

TART 

SEL I SEAR 

w 

G 1 

TROTTA 

CHRISTENSEN 

1 0 

0  4 

TROT T A 

CHRISTENSEN 

1  1 
12 

13 

14 

15 

16 
17 


9 

04 

BA  I N 

NGUYEN 

1 0 

02 

NGUYEN 

BAIN 

1 1 

R1 

MQMAHAM 

MEMKE 

12 

R2 

MONAHAN 

MENKE 

13 

02 

SAVOYE 

14 

□  1 

SAVOYE 

15 

02 

MCCLELLEN 

1  6 

03 

BA  I N 

TABLE  2-2  con ' t 


TOES  3 

9 

R2 

MOORE 

10 

R1 

MOORE 

11 

03 

NGUYEN 

12 

Oi 

NGUYEN 

13 

R1 

KARTY 

14 

R3 

KARTY 

15 

R3 

SCHMITZ 

MENKE 

16 

R4 

SCHMITZ 

MENKE 

17 

WED  4 

B 

B2 

TROTTA 

9 

02 

WILSON 

10 

01 

WILSON 

1  1 

01 

12 

X 

13 

V 

14 

X 

15 

X 

16 

y 

1 7 

X 

THUR  5 

El 

G3 

TROTTA 

9 

R4 

K.A!’’;  i  Y 

10 

R2 

KARTY 

1 1 

R4 

MOORE 

12 

R3 

MOORE 

13 

02 

ORNDORF-F 

14 

03 

CAF'PEL 

15 

F<  1 

SCHMITZ 

16 

R2 

SCHMITZ 

17 

FRI  6 

9 

X 

10 

X 

1  1 

X 

12 

V 

13 

X 

14 

y 

15 

V 

1 6 

X 

17 

X 

MON  9 

9 

02 

CAPPEL 

10 

□  1 

CAPPEL 

BAIN 

1  1 

G2 

1  2 

G3 

13 

R3 

MONAHAN 

14 

R4 

MONAHAN 

15 

X 

16 

X 

17 

X 

TOES  iO 

9 

G3 

CHAMPION 

KEFilR 

10 

G2 

CHAMPION 

KERR 

1 1 

R3 

JAMIL 

12 

01 

MCCLELLi 

13 

03 

WILSON 

THOMPSON 

THOMPSON 


RAINES 

RAINES 


BAKER 

BAKER 


THOMPSON 
0  ■'  CONNDE; 
0  ■■  CONNOR 


LABONTE 

LABONTE 


CHRISTENSEN 

CHRISTENSEN 


E'AKEF'! 


TABLE  2-2  con ' t 


WED  11 

o 

/ 

61 

10 

64 

11 

R2 

12 

R1 

13 

62 

14 

61 

KERR 

15 

04 

OWENS 

QUINN 

16 

02 

OWENS 

QUINN 

a 

03 

QUINN 

THURS  12 

9 

01 

OWENS 

0 ' CONNOR 

QUINN 

10 

R1 

PERSCHAU 

1 1 

R2 

PERSCHALI 

12 

R2 

BRUSH 

1  ■:r 

X 

03 

OWENS 

i  4 

G1 

POOLE 

15 

G4 

KERR 

POOLE 

1 6 

17 

o 

04 

0 " CONNOR 

CARPEL 

FR I  13 

9 

02 

RYAN 

OLDEN 

10 

03 

F-  Y  AN 

OLDEN 

1  1 

R4 

BRUSH 

THOMPSON 

PERSCHAU 

12 

R3 

BRUSH 

PERSCHAU 

13 

01 

□RNDORFF 

1  '1 

R1 

RAINES 

GRAY 

NGUYEN 

15 

R2 

RAINES 

OR  A  Y 

NGUYEN 

16 

R1 

BRUSH 

17 

WED  la 

9 

R3 

GRAY 

BAE 

1 0 

1  1 

12 

1  3 

14 

R4 

GRAY 

BAE 

15 

16 

17 

THLJR  26 

9 

10 

Ri 

BAE 

1  1 

R2 

BAE 

i 

X  ^ 

13 

14 

15 

16 

17 
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Video  equipment  for  viewing  the  test  tapes  was  provided  by  Sony 
Electronics,  consisting  of  a  Broadcast  Television  Monitor,  Model  BVM1911,  and  a 
Betacam  SP  Tape  Player,  Model  PVW2600.  Calibration  of  the  video  equipment 
was  done  each  day  prior  to  scheduled  viewing,  consisting  of  checks  of  the  tape 
player  output  amplitude  (white  =  714  IEEE  units),  monitor  white  balance  (6500 
degrees  K),  monitor  peak  white  brightness  (70.0  cd/square  meter),  and  monitor 
chroma  balance.  The  initial  setup  of  the  video  equipment  was  facilitated  by  use  of 
a  Betacam  Video  Test  Signal  Generator,  Tektronix  Model  TSG130A,  and  a  Video 
Waveform  Monitor,  Tektronix  Model  WFM300A.  Monitoring  of  all  tape  plays  with 
the  WFM300A  was  done  by  the  test  conductor  to  assure  normal  video  production. 

2.5.2  Color  Perception 

Following  the  completion  of  tape  viewing,  vision  screening  checks  were 
made,  if  required.  Color  perception  checks  were  made  with  use  of  "Pseudo- 
Isochromatic  Plates  for  Color  Perception,"  made  by  Richmond  Products,  1021 
Rogers  Circle,  Boca  Raton,  Florida,  which  test  for  red-green  color  defects.  These 
color  checks  were  performed  in  the  "Expert  Viewing  Room"  with  the  room 
illumination  at  maximum  level.  The  room  illumination  is  D65,  which  is  required  for 
viewing  the  color  plates.  A  score  sheet  was  marked  by  the  viewer  and  retained  for 
recording  purposes.  Normal  color  vision  was  considered  to  be  missing  no  more 
than  4  of  the  14  color  plates.  Of  the  38  viewers  tested,  two  did  not  demonstrate 
normal  color  vision. 

2.5.3  Visual  Acuity 

Visual  acuity  checks  were  next  made,  if  required,  prior  to  releasing  the 
viewer.  These  were  done  with  a  normal  eye  chart  reduced  in  size  for  viewing  at 
5.5  feet,  as  distributed  by  NTIA.  Viewers  were  expected  to  exhibit  no  errors  on 
line  7.  For  the  first  two  days,  viewers  were  tested  in  the  standing  position. 

Several  viewers  confused  the  first  character  of  line  7,  recognizing  it  as  a  "P"  rather 
than  an  "F".  Viewers  were  then  checked  in  a  sitting  position,  with  consistently 
better  results  for  the  remaining  days  of  the  test.  When  possible,  viewers  were 
later  rechecked  in  the  sitting  position  and  generally  passed  the  test.  Because  it 
was  not  possible  to  recheck  all  viewers,  the  SUMMARY  program  was  modified  to 
not  disqualify  viewers  that  had  no  errors  only  on  line  6,  since  it  was  felt  that  all  the 
viewers  had  normal  visual  acuity. 
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2.5.4  Vision  Summary 


Table  2-1  shows  the  demographic  data  accumulated  on  the  viewers.  Under 
EYE  TEST,  the  first  number  indicates  the  line  of  the  acuity  chart  that  was  read 
with  no  errors;  the  second  number  indicates  the  number  of  color  charts  missed. 
This  shows  that  5  viewers  only  read  line  6  of  the  acuity  chart  with  no  errors  and 
that  6  viewers  read  all  color  plates  correctly. 

2.5.5  Viewing  Position 

It  was  not  realized  that  it  was  required  that  each  viewer  be  assigned  the 
same  seat  at  each  session.  This  requirement  is  stated  only  in  Appendix  G  of  the 
Subjective  Test  Plan.  Adhering  to  this  requirement  would  have  added  to  the 
scheduling  difficulties  already  presented  by  the  availability  of  viewers  and 
maintaining  a  team  schedule.  The  SUMMARY  program  would  not  accept  the 
absence  of  a  single  seat  designation  in  the  demographic  data.  Therefore,  a  seat 
was  inserted  for  this  item  (generally  that  of  the  majority  or  the  first  session).  In 
addition,  the  WordPerfect  file  "SEATS"  was  generated  which  shows  the  seats 
used  by  each  viewer  in  each  session,  shown  in  Table  2-3.  This  file  should  be  used 
if  statistical  tests  on  the  effect  of  viewing  position  are  performed. 

2.5.6  Viewing  Order 

Because  of  scheduling  difficulties,  it  was  also  not  possible  to  schedule  the 
planned  order  for  viewing  tapes.  Note  that  for  each  team,  there  are  24  possible 
sequences  of  the  four  tapes.  Four  of  these  sequences  were  assigned  to  each  of 
the  three  LABs;  so,  at  most,  only  12  of  the  24  orders  were  scheduled.  See 
Table  2-4  for  the  distribution  of  viewing  sequences  for  the  three  LABs  and  for  the 
accomplished  viewing  sequences  done  by  the  viewers  conducted  by  LAB  X. 

Letters  X,  Y,  and  Z  indicate  the  planned  orders.  The  numbers  underneath  indicate 
the  total  number  of  viewers  at  LAB  X  for  each  team.  Those  numbers  in  brackets 
indicate  the  number  of  qualified  viewers  if  less  than  the  total  viewers.  Numbers 
not  under  an  X  indicate  viewers  whose  sequence  was  not  according  to  the 
schedule. 
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TABLE  2-3  ACTUAL  SEATS  USED  BY  VIEWERS 


The  following  are  the  seats  used  by  the  viewers.  They  are  listed  in  the 
chronological  order  in  which  they  viewed  the  tapes,  not  by  the  tape  number. 


VIEWER  SESSION 
12  3  4 


OlX 

02x 

03X 

04x 

05x 

06x 

07x 

08x 

09x 

lOx 

llx 

12x 

13x 

14X 

15x 

16x 

17x 

18x 

19X 

20x 

21x 

22x 

23x 

24x 

25x 

26x 

27x 

28x 

29x 

30x 

3  lx 
32x 
33x 
34X 
35X 
36x 
37X 
38X 
39x 
40x 

4  lx 


c  c  c  c 
r  r  c  c 
c  c  r  1 


r  r  r  1 
1  1  c  c 
1  1  c  c 
r  r  c  c 
r  r  r  r 
1  c  c  c 
c  c  c  c 

c  c  r  r 
r  r  1  r 

cell 


r  r  r  r 
c  c  c  c 
c  1  1  c 
r  r  r  r 
r  r  1  1 
c  c  c  r 
1  r  r  c 
1  1  c  c 
c  r  c  c 
c  c  c  c 
c  r  r  c 
1  1  c  c 
cell 
111c 
1111 
r  r  c  c 
r  1  c  c 
lire 
c  c  c  c 


r  r  c  c 
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TABLE  2-4  DISTRIBUTION  OF  VIEWING  SEQUENCES 


1234  1243 

1324 

1342 

1423 

1432 

2134 

2143 

2314 

2341 

2413 

2431 

RED 

X 

X 

y 

X 

yy 

Z 

Z 

3  1 

2 

1(0) 

1(0) 

1 

1 

GREEN 

z 

y 

X 

X 

X 

Z 

Z 

1 

4 

2 

ORANGE 

y 

Z 

X 

y 

2 

1 

3124  3142 

3214 

3241 

3412 

3421 

4123 

4132 

4213 

4231 

4312 

4321 

RED 

yz 

X 

z 

3(2) 

GREEN 

y  X  y  y  z 

3 

ORANGE 

y  z  xyz  xzx 

111  3(2)  3(2) 


Five  otherwise  qualified  viewers  (15x,  22x,  24x,  26x,  and  37x)  viewed  the 
tapes  in  an  order  that  was  not  in  the  original  plan.  Only  one  of  these  unplanned 
orders  coincided  with  that  of  the  same  team  of  another  LAB,  so  diversity  of  order 
was  increased. 


2.6  Data  Collection 


Original  test  data  sheets  were  marked  by  the  viewer,  showing  the  scheduled 
time  and  date  as  well  as  the  actual  time  and  date  of  each  tape  play.  Each  scene  of 
each  tape  has  space  assigned  for  the  viewer  to  mark  one  of  five  opinion  grades  as 
shown  in  the  test  plan.  Included  in  each  tape  sequence  are  nulls  and  repeats.  A 
null  is  a  scene  with  no  degradation;  a  repeat  is  an  identical,  same  scene/HRC 
combination  for  the  second  time  in  a  session.  These  nulls  and  repeats  were 
checked  following  the  completion  of  the  viewing  period  which  identified  probable 
invalid  data  to  facilitate  subsequent  viewer  scheduling.  Original  data  sheets  are  on 
file  at  Delta  Information  Systems. 
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2.6.1  Missed  Repeats 


Of  the  9,030  scene  combinations  to  be  marked  by  all  viewers,  1 1  were 
missed,  either  because  no  score,  or  more  than  one  score,  was  marked.  For  two 
viewers,  09x  and  27x,  one  of  the  missed  scores  was  the  second  repeat. 

According  to  the  Subjective  Test  Plan,  these  viewers  should  be  disqualified 
because  they  missed  a  quality  check  combination.  However,  27x  scored  every  null 
and  all  other  repeats  correctly.  The  score  on  the  first  repeat  was  4,  so  only  a  1  on 
the  second  repeat  would  be  disqualifying  otherwise.  The  repeat  in  question  was 
not  marked,  and  the  previous  scene  had  two  scores:  3  and  5.  The  situation  for 
09x  was  similar  in  that  every  null  was  correct,  and  the  other  repeats  were  scores 
differing  no  more  than  1 .  The  first  repeat  score  was  4,  and  the  second  repeat  was 
scored  both  3  and  5.  It  was  felt  that  this  should  not  disqualify  the  viewer  since 
the  repeat  score  does  not  count  towards  the  result.  Therefore,  the  SUMMARY 
program  was  modified  so  that  these  viewers  were  not  disqualified. 

2.6.2  Missed  Scores 

For  another  viewer,  lOx,  one  score  was  missed  on  each  of  three  tapes. 
According  to  the  Subjective  Test  Plan,  this  viewer  should  be  disqualified  for 
missing  more  than  two  scores.  Viewer  09x  had  a  similar  situation  in  missing  two 
scores  on  one  and  one  score  on  another  tape.  Apparently,  the  SUMMARY 
program  does  not  check  for  this. 

2.6.3  Null  Checks 

Five  viewers  (07x,  30x,  31  x,  33x,  and  36x)  were  disqualified  because  of  the 
null  check.  Both  07x  and  36x  missed  two  nulls,  with  36x  giving  a  score  of  2  to  a 
null.  The  scenes  missed  were  t  (three  times),  s  (two  times),  c,  and  h.  No  viewers 
failed  the  repeat  check,  and  only  8  of  140  repeat  had  a  difference  as  large  as  2. 

2.7  Data  Reduction 

Data  sheets  collected  during  the  viewing  series  were  entered  into  a  data  file 
and  verified  by  a  second  data  entry  with  comparison.  Seventeen  entry  errors  of 
the  first  data  files  were  corrected  by  the  check  method.  The  scores  of  the  38 
viewers  that  were  processed  in  the  viewing  program  were  reduced  by  5  failing  the 
accuracy  checks,  2  incompletes  due  to  lack  of  desired  color  perception,  and  1  that 
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was  not  able  to  complete  the  four-tape  series.  The  30  satisfactory  scores  were 
equally  distributed  among  the  three  color  teams. 

These  scores  were  processed  by  the  SUMMARY  program  developed  by  the 
National  Telecommunications  &  Information  Administration.  This  program 
calculates  the  Mean  Opinion  Score  (MOS)  of  each  scene  (a  to  y)  for  all  HRCs  (1  to 
25).  Also  calculated  are  SD  (estimate  of  standard  deviation  of  observer 
population)  and  SE  (estimate  of  standard  error  of  mean)  for  all.  The  Maximum 
(largest  observed  opinion  score)  and  Minimum  (smallest  observed  opinion  score) 
scores  are  also  shown  for  all.  This  data  is  shown  in  Table  2-5. 

MOS  for  Scene/HRC  of  another  LAB  is  shown  in  Appendix  C  for  LAB  Z 
(Department  of  Commerce/National  Telecommunications  &  Information 
Administration/Institute  for  Telecommunications  Sciences). 

2.8  Final  Data  Details 

All  test  participants  of  teams  viewed  and  scored  scenes  processed  by  HRC 
20,  as  well  as  scenes  d  and  s  processed  by  various  HRCs.  Figure  2.2  shows  the 
scores  of  the  NCS/DIS  program  compared  with  scores  reported  from  the  NTIA 
program  for  HRC  20.  Figures  2.3  and  2.4  show  comparisons  of  scores  also 
reported  by  the  same  programs  for  scenes  d  and  s.  Figures  2.5,  2.6,  and  2.7 
show  the  same  data  but  comparing  LAB  Y  (GTE)  against  the  Delta  data. 

The  plot  of  MOS  comparing  the  results  for  two  LABs  would  be  in  complete 
agreement  when  the  data  point  falls  on  the  dotted  line.  For  nearly  all  scenes,  the 
MOS  LAB-to-LAB  agreement  is  quite  close. 
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TABLE  2-5  SUMMARY  OF  RESULTS 


HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

1 

MEAN 

4.90 

5.00 

4.80 

4.90 

5.00 

4.90 

4.80 

5.00 

4.80 

4.90 

5.00 

5.00 

5.00 

S.D. 

0.32 

0.00 

0.42 

0.32 

0.00 

0.32 

0.42 

0.00 

0.63 

0.32 

0.00 

0.00 

0.00 

S.E. 

0.10 

0.00 

0.13 

0.10 

0.00 

0.10 

0.13 

0.00 

0.20 

0.10 

0.00 

0.00 

0.00 

MAX 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

MIN 

4 

5 

4 

4 

5 

4 

4 

5 

3 

4 

5 

5 

5 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

2 

MEAN 

4.80 

4.20 

4.20 

4.40 

4.70 

4.20 

4.60 

4.20 

4.50 

4.10 

4.60 

4.80 

4.67 

S.D. 

0.42 

0.63 

0.92 

0.70 

0.48 

0.42 

0.52 

0.63 

0.53 

0.57 

0.52 

0.42 

0.50 

S.E. 

0.13 

0.20 

0.29 

0.22 

0.15 

0.13 

0.16 

0.20 

0.17 

0.18 

0.16 

0.13 

0.17 

MAX 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

MIN 

4 

3 

3 

3 

4 

4 

4 

3 

4 

3 

4 

4 

4 

HRC 

a 

Bl 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

3 

MEAN 

4.90 

4.70 

4.80 

4.80 

4.90 

4.80 

4.90 

4.70 

4.90 

4.90 

4.90 

4.50 

4.90 

S.D. 

0.32 

0.67 

0.42 

0.42 

0.32 

0.42 

0.32 

0.48 

0.32 

0.32 

0.32 

0.53 

0.32 

S.E. 

0.10 

0.21 

0.13 

0.13 

0.10 

0.13 

0.10 

0.15 

0.10 

0.10 

0.10 

0.17 

0.10 

MAX 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

MIN 

4 

3 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

HRC 

a 

b 

C 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

4 

MEAN 

3.00 

2.60 

2.45 

2.45 

2.15 

3.25 

2.85 

2.25 

1.45 

3.10 

3.15 

3.55 

1.85 

S.D. 

0.79 

0.94 

1.05 

0.89 

1.04 

0.72 

0.81 

0.91 

1.00 

0.85 

0.99 

0.69 

0.81 

S.E. 

0.18 

0.21 

0.23 

0.20 

0.23 

0.16 

0.18 

0.20 

0.22 

0.19 

0.22 

0.15 

0.18 

MAX 

4 

4 

5 

4 

4 

4 

4 

4 

5 

4 

5 

5 

3 

MIN 

2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 

1 

HRC 

a 

■ 

C 

d 

e 

f 

g 

■ 

n| 

j 

k 

1 

m 

5 

MEAN 

3.60 

3.20 

3.30 

3.20 

3.40 

4.10 

3.20 

2.90 

1.60 

3.80 

3.90 

4.10 

2.90 

S.D. 

0.70 

0.79 

0.95 

0.79 

0.97 

0.57 

0.92 

0.74 

0.52 

0.63 

0.88 

0.32 

0.57 

S.E. 

0.22 

0.25 

0.30 

0.25 

0.31 

0.18 

0.29 

0.23 

0.16 

0.20 

0.28 

0.10 

0.18 

MAX 

5 

4 

5 

4 

5 

5 

4 

4 

2 

5 

5 

5 

4 

MIN 

3 

2 

2 

2 

2 

3 

2 

2 

1 

3 

3 

4 

2 

2 


16 


TABLE  2-5  con't 


HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

6 

MEAN 

3.70 

2.50 

2.90 

2.50 

2.20 

4.20 

3.00 

1.70 

1.20 

2.70 

3.70 

3.70 

1.70 

S.D. 

0.82 

0.71 

0.99 

1.18 

1.14 

0.42 

0.82 

0.67 

0.63 

0.67 

0.48 

0.67 

0.48 

S.E. 

0.26 

0.22 

0.31 

0.37 

0.36 

0.13 

0.26 

0.21 

0.20 

0.21 

0.15 

0.21 

0.15 

MAX 

5 

4 

4 

5 

4 

5 

4 

3 

3 

4 

4 

5 

2 

MIN 

2 

2 

1 

1 

1 

4 

2 

1 

1 

2 

3 

3 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

7 

MEAN 

3.70 

3.00 

2.90 

3.20 

3.44 

3.60 

3.20 

3.10 

2.20 

3.70 

3.70 

3.60 

2.30 

S.D. 

0.82 

0.82 

0.99 

1.14 

0.88 

0.84 

0.92 

0.74 

0.79 

0.67 

0.82 

0.84 

0.95 

S.E. 

0.26 

0.26 

0.31 

0.36 

0.29 

0.27 

0.29 

0.23 

0.25 

0.21 

0.26 

0.27 

0.30 

MAX 

5 

4 

4 

5 

4 

5 

4 

4 

3 

4 

5 

5 

4 

MIN 

2 

1 

1 

1 

2 

2 

1 

2 

1 

2 

2 

2 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

8 

MEAN 

4.00 

3.50 

2.70 

3.60 

3.40 

3.70 

3.80 

3.00 

2.50 

4.20 

3.90 

4.20 

3.50 

S.D. 

0.67 

0.97 

1.06 

0.84 

1.07 

0.95 

0.63 

1.15 

0.71 

0.92 

0.88 

0.63 

0.85 

S.E. 

0.21 

0.31 

0.33 

0.27 

0.34 

0.30 

0.20 

0.37 

0.22 

0.29 

0.28 

0.20 

0.27 

MAX 

5 

5 

4 

5 

4 

5 

5 

4 

4 

5 

5 

5 

4 

MIN 

3 

2 

1 

2 

1 

2 

3 

1 

2 

2 

2 

3 

2 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

9 

MEAN 

4.70 

3.30 

2 . 60 

3.40 

3.50 

4.50 

4.00 

2.60 

2.20 

3.90 

4.20 

4.10 

2.60 

S.D. 

0.48 

0.67 

0.52 

0.84 

0.53 

0.53 

0.67 

0.84 

0.79 

0.57 

0.63 

0.74 

0.70 

S.E. 

0.15 

0.21 

0.16 

0.27 

0.17 

0.17 

0.21 

0.27 

0.25 

0.18 

0.20 

0.23 

0.22 

MAX 

5 

4 

3 

4 

4 

5 

5 

4 

3 

5 

5 

5 

4 

MIN 

4 

2 

2 

2 

3 

4 

3 

1 

1 

3 

3 

3 

2 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

10 

MEAN 

4.90 

4.00 

3.20 

3.67 

3.60 

4.90 

4.40 

3.40 

2.80 

4.60 

4.80 

4.60 

4.00 

S.D. 

0.32 

0.67 

0.63 

0.50 

0.70 

0.32 

0.52 

0.70 

0.63 

0.52 

0.42 

0.52 

0.50 

S.E. 

0.10 

0.21 

0.20 

0.17 

0.22 

0.10 

0.16 

0.22 

0.20 

0.16 

0.13 

0.16 

0.17 

MAX 

5 

5 

4 

4 

5 

5 

5 

4 

4 

5 

5 

5 

5 

MIN 

4 

3 

2 

3 

3 

4 

4 

2 

2 

4 

4 

4 

3 

2 


17 


TABLE  2-5  con't 


HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

11 

MEAN 

3.00 

3.10 

1.70 

2.20 

1.80 

3.10 

2.20 

1.60 

1.40 

1.70 

2.60 

2.20 

1.40 

S.D. 

0.82 

0.57 

0.67 

0.79 

0.79 

0.88 

0.79 

0.84 

0.70 

0.67 

0.70 

0.92 

0.70 

S.E. 

0.26 

0.18 

0.21 

0.25 

0.25 

0.28 

0.25 

0.27 

0.22 

0.21 

0.22 

0.29 

0.22 

MAX 

4 

4 

3 

3 

3 

4 

4 

3 

3 

3 

4 

4 

3 

MIN 

2 

2 

1 

1 

1 

2 

1 

1 

1 

1 

2 

1 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

12 

MEAN 

2.60 

1.90 

1.70 

1.90 

1.50 

2.90 

2.50 

1.90 

1.30 

2.20 

2.30 

3.00 

1.30 

S.D. 

0.84 

0.74 

1.06 

0.88 

0.71 

0.99 

1.08 

0.88 

0.48 

0.63 

0.95 

0.82 

0.48 

S.E. 

0.27 

0.23 

0.33 

0.28 

0.22 

0.31 

0.34 

0.28 

0.15 

0.20 

0.30 

0.26 

0.15 

MAX 

4 

3 

4 

3 

3 

4 

4 

3 

2 

4 

4 

4 

2 

MIN 

1 

1 

1 

1 

1 

2 

1 

1 

1 

2 

1 

2 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

in 

13 

MEAN 

1.90 

1.60 

1.60 

2.20 

1.80 

2.60 

2.20 

2.20 

1.80 

2.40 

2.60 

3.00 

2.00 

S.D. 

0.74 

0.84 

0.52 

0.79 

0.79 

0.97 

1.23 

1.03 

0.92 

0.84 

1.07 

0.94 

0.67 

S.E. 

0.23 

0.27 

0.16 

0.25 

0.25 

0.31 

0.39 

0.33 

0.29 

0.27 

0.34 

0.30 

0.21 

MAX 

3 

3 

2 

3 

3 

4 

4 

4 

3 

4 

4 

4 

3 

MIN 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

HRC 

a 

Bii 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

14 

MEAN 

2.10 

2.70 

2.10 

2.40 

1.90 

2.80 

1.90 

2.40 

2.20 

3.30 

3.00 

3.30 

3.20 

S.D. 

0.88 

1.34 

0.99 

1.07 

0.88 

0.92 

0.88 

0.97 

0.63 

0.67 

0.82 

0.95 

0.63 

S.E. 

0.28 

0.42 

0.31 

0.34 

0.28 

0.29 

0.28 

0.31 

0.20 

0.21 

0.26 

0.30 

0.20 

MAX 

4 

5 

4 

4 

3 

4 

3 

4 

3 

4 

4 

5 

4 

MIN 

1 

1 

1 

1 

1 

2 

1 

1 

1 

2 

2 

2 

2 

HRC 

a 

Bii 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

15 

MEAN 

2.80 

2.40 

1.55 

2.10 

1.35 

3.10 

2.50 

1.95 

1.55 

2.05 

2.95 

2.40 

1.70 

S.D. 

0.77 

0.94 

0.76 

0.85 

0.67 

0.97 

0.89 

0.83 

0.69 

0.83 

0.89 

0.94 

0.66 

S.E. 

0.17 

0.21 

0.17 

0.19 

0.15 

0.22 

0.20 

0.18 

0.15 

0.18 

0.20 

0.21 

0.15 

MAX 

4 

4 

3 

4 

3 

5 

4 

3 

3 

4 

4 

4 

3 

MIN 

2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 


18 


TABLE  2-5  con't 


HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

16 

MEAN 

3.40 

3.20 

2.90 

2.56 

1.90 

3.30 

2.40 

2.30 

1.10 

1.80 

2.60 

3.30 

1.40 

S.D. 

0.97 

0.63 

1.37 

0.88 

0.74 

1.06 

0.70 

0.95 

0.32 

0.63 

0.70 

0.82 

0.52 

S.E. 

0.31 

0.20 

0.43 

0.29 

0.23 

0.33 

0.22 

0.30 

0.10 

0.20 

0.22 

0.26 

0.16 

MAX 

4 

4 

5 

4 

3 

5 

4 

4 

2 

3 

4 

5 

2 

MIN 

1 

2 

1 

1 

1 

2 

2 

1 

1 

1 

2 

2 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

17 

MEAN 

3.65 

3.05 

2.45 

2.65 

1.95 

3.70 

3.00 

1.95 

1.65 

2.40 

3.05 

3.15 

1.70 

S.D. 

0.67 

0.83 

1.05 

1.04 

0.89 

0.73 

0.73 

1.00 

0.88 

0.99 

0.69 

0.81 

0.92 

S.E. 

0.15 

0.18 

0.23 

0.23 

0.20 

0.16 

0.16 

0.22 

0.20 

0.22 

0.15 

0.18 

0.21 

MAX 

5 

5 

4 

5 

4 

5 

4 

4 

4 

5 

4 

5 

4 

MIN 

2 

2 

1 

1 

1 

2 

2 

1 

1 

1 

2 

2 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

18 

MEAN 

4.00 

2.90 

2.60 

2.70 

2.60 

4.30 

3.40 

2.40 

1.40 

2.90 

3.60 

3.80 

1.40 

S.D. 

0.67 

0.74 

1.07 

0.67 

0.70 

0.48 

0.70 

0.84 

0.70 

0.88 

0.52 

0.63 

0.52 

S.E. 

0.21 

0.23 

0.34 

0.21 

0.22 

0.15 

0.22 

0.27 

0.22 

0.28 

0.16 

0.20 

0.16 

MAX 

5 

4 

5 

4 

4 

5 

4 

3 

3 

4 

4 

5 

2 

MIN 

3 

2 

1 

2 

2 

4 

2 

1 

1 

2 

3 

3 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

19 

MEAN 

3.70 

2.50 

2.10 

2.80 

2.60 

3.60 

3.50 

2.60 

2.60 

2.70 

3.40 

3.80 

2.40 

S.D. 

0.67 

0.71 

1.10 

0.92 

0.84 

0.97 

0.71 

1.07 

0.97 

0.82 

1.07 

0.63 

0.97 

S.E. 

0.21 

0.22 

0.35 

0.29 

0.27 

0.31 

0.22 

0.34 

0.31 

0.26 

0.34 

0.20 

0.31 

MAX 

5 

3 

4 

4 

4 

5 

4 

4 

4 

4 

4 

5 

4 

MIN 

3 

1 

1 

1 

1 

2 

2 

1 

1 

1 

1 

3 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

20 

MEAN 

4.47 

3.50 

2.77 

3.37 

3.50 

4.30 

4.00 

2.67 

2.23 

3.43 

4.13 

4.17 

2.50 

S.D. 

0.63 

0.73 

0.94 

0.76 

0.63 

0.75 

0.59 

0.96 

1.01 

0.86 

0.90 

0.65 

0.90 

S.E. 

0.11 

0.13 

0.17 

0.14 

0.11 

0.14 

0.11 

0.18 

0.18 

0.16 

0.16 

0.12 

0.16 

MAX 

5 

4 

5 

4 

4 

5 

5 

4 

4 

5 

5 

5 

4 

MIN 

3 

1 

1 

1 

2 

2 

2 

1 

1 

1 

1 

3 

1 

2 


19 


TABLE  2-5  con't. 


HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

21 

MEAN 

3.80 

2.90 

2.90 

2.50 

2.40 

3.20 

3.40 

2.90 

1.40 

2.60 

3.40 

3.20 

2.40 

S.D. 

0.63 

0.74 

0.57 

0.71 

0.84 

0.79 

0.84 

0.74 

0.52 

0.70 

0.52 

0.63 

0.52 

S.E. 

0.20 

0.23 

0.18 

0.22 

0.27 

0.25 

0.27 

0.23 

0.16 

0.22 

0.16 

0.20 

0.16 

MAX 

5 

4 

4 

4 

4 

4 

5 

4 

2 

4 

4 

4 

3 

MIN 

3 

2 

2 

2 

1 

2 

2 

2 

1 

2 

3 

2 

2 

HRC 

a 

tm 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

22 

MEAN 

4.20 

3.00 

2.80 

3.90 

3.80 

4.10 

4.20 

3.30 

2.80 

4.40 

4.30 

4.20 

3.20 

S.D. 

0.92 

1.05 

1.32 

0.88 

1.03 

0.99 

0.79 

1.16 

0.63 

0.70 

0.82 

0.92 

1.03 

S.E. 

0.29 

0.33 

0.42 

0.28 

0.33 

0.31 

0.25 

0.37 

0.20 

0.22 

0.26 

0.29 

0.33 

MAX 

5 

4 

4 

5 

5 

5 

5 

5 

4 

5 

5 

5 

4 

MIN 

2 

1 

1 

2 

2 

2 

3 

1 

2 

3 

3 

2 

1 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

m 

23 

MEAN 

4.10 

3.60 

3.20 

3.20 

3.50 

3.80 

4.40 

1.70 

1.70 

4.30 

4.80 

4.20 

2.50 

S.D. 

0.88 

0.70 

0.63 

1.03 

0.97 

0.79 

0.70 

0.82 

0.67 

0.48 

0.42 

0.63 

0.85 

S.E. 

0.28 

0.22 

0.20 

0.33 

0.31 

0.25 

0.22 

0.26 

0.21 

0.15 

0.13 

0.20 

0.27 

MAX 

5 

4 

4 

4 

5 

5 

5 

3 

3 

5 

5 

5 

4 

MIN 

2 

2 

2 

1 

2 

2 

3 

1 

1 

4 

4 

3 

1 

HRC 

a 

PI 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

Itl 

24 

MEAN 

4.40 

4.00 

3.70 

3.90 

3.70 

4.30 

4.40 

3.60 

3.30 

4.50 

4.30 

3.60 

3.70 

S.D. 

0.84 

0.94 

0.67 

1.10 

1.06 

1.25 

0.84 

1.07 

1.06 

0.71 

1.25 

0.97 

0.95 

S.E. 

0.27 

0.30 

0.21 

0.35 

0.33 

0.40 

0.27 

0.34 

0.33 

0.22 

0.40 

0.31 

0.30 

MAX 

5 

5 

4 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

MIN 

3 

2 

2 

2 

2 

2 

3 

2 

2 

3 

1 

2 

2 

HRC 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

Til 

25 

MEAN 

4.30 

3.50 

3.20 

3.60 

3.90 

4.40 

3.80 

3.30 

3.20 

4.20 

4.30 

4.40 

3.80 

S.D. 

0.48 

0.53 

0.79 

0.70 

0.57 

0.70 

0.79 

0.82 

0.79 

0.42 

0.67 

0.52 

0.63 

S.E. 

0.15 

0.17 

0.25 

0.22 

0.18 

0.22 

0.25 

0.26 

0.25 

0.13 

0.21 

0.16 

0.20 

MAX 

5 

4 

4 

5 

5 

5 

5 

4 

4 

5 

5 

5 

5 

MIN 

4 

3 

2 

3 

3 

3 

3 

2 

2 

4 

3 

4 

3 

2 


20 


TABLE  2-5  con't 


HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

1 

MEAN 

4.60 

4.90 

4.70 

5.00 

5.00 

5.00 

5.00 

4.90 

4.70 

5.00 

4.90 

5.00 

S.D. 

1.26 

0.32 

0.48 

0.00 

0.00 

0.00 

0.00 

0.32 

0.48 

0.00 

0.32 

0.00 

S.E. 

0.40 

0.10 

0.15 

0.00 

0.00 

0.00 

0.00 

0.10 

0.15 

0.00 

0.10 

0.00 

MAX 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

MIN 

1 

4 

4 

5 

5 

5 

5 

4 

4 

5 

4 

5 

HRC 

n 

o 

P 

q 

r 

S 

t 

u 

V 

W 

X 

y 

2 

MEAN 

5.00 

4.60 

4.60 

4.90 

4.50 

4.10 

4.00 

4.40 

4.30 

4.40 

4.60 

4.60 

S.D. 

0.00 

0.52 

0.52 

0.32 

0.53 

0.57 

0.82 

0.52 

0.82 

0.52 

0.52 

0.70 

S.E. 

0.00 

0.16 

0.16 

0.10 

0.17 

0.18 

0.26 

0.16 

0.26 

0.16 

0.16 

0.22 

MAX 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

MIN 

5 

4 

4 

4 

4 

3 

3 

4 

3 

4 

4 

3 

HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

3 

MEAN 

4.70 

4.70 

4.40 

4.90 

4.70 

4.70 

4.90 

4.70 

4.60 

4.70 

4.70 

4.50 

S.D. 

0.67 

0.67 

0.84 

0.32 

0.48 

0.48 

0.32 

0.67 

0.52 

0.48 

0.67 

0.53 

S.E. 

0.21 

0.21 

0.27 

0.10 

0.15 

0.15 

0.10 

0.21 

0.16 

0.15 

0.21 

0.17 

MAX 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

MIN 

3 

3 

3 

4 

4 

4 

4 

3 

4 

4 

3 

4 

HRC 

n 

O 

P 

q 

r 

s 

t 

u 

V 

W 

X 

y 

4 

MEAN 

1.65 

2.55 

2.40 

1.70 

1.55 

1.60 

1.75 

2.90 

2.30 

2.75 

2.65 

1.60 

S.D. 

0.67 

0.69 

0.88 

0.66 

0.76 

1.05 

0.85 

0.97 

0.98 

0.79 

0.88 

0.88 

S.E. 

0.15 

0.15 

0.20 

0.15 

0.17 

0.23 

0.19 

0.22 

0.22 

0.18 

0.20 

0.20 

MAX 

3 

4 

4 

3 

3 

5 

4 

4 

4 

4 

4 

4 

MIN 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

5 

MEAN 

2.60 

3.20 

3.40 

2.20 

3.20 

1.20 

2.50 

3.90 

3.30 

3.40 

3.10 

3.00 

S.D. 

0.70 

0.63 

0.84 

0.79 

0.79 

0.42 

0.71 

0.57 

0.67 

0.84 

0.74 

0.82 

S.E. 

0.22 

0.20 

0.27 

0.25 

0.25 

0.13 

0.22 

0.18 

0.21 

0.27 

0.23 

0.26 

MAX 

4 

4 

4 

3 

5 

2 

3 

5 

4 

5 

4 

4 

MIN 

2 

2 

2 

1 

2 

1 

1 

3 

2 

2 

2 

2 

2 


21 


TABLE  2-5  con't 


HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

6 

MEAN 

1.70 

2.30 

2.90 

1.20 

2.10 

1.20 

2.00 

3.80 

2.00 

2.70 

2.90 

2.20 

S.D. 

0.67 

0.95 

0.88 

0.42 

0.32 

0.42 

0.94 

0.42 

0.82 

0.82 

0.99 

1.03 

S.E. 

0.21 

0.30 

0.28 

0.13 

0.10 

0.13 

0.30 

0.13 

0.26 

0.26 

0.31 

0.33 

MAX 

3 

4 

4 

2 

3 

2 

4 

4 

3 

4 

5 

4 

MIN 

1 

1 

2 

1 

2 

1 

1 

3 

1 

2 

2 

1 

HRC 

n 

mm 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

7 

MEAN 

2.70 

3.40 

3.50 

2.90 

3.10 

2.30 

2.67 

3.60 

2.80 

3.20 

3.50 

2.90 

S.D. 

1.16 

0.97 

0.71 

1.10 

1.10 

0.95 

1.12 

0.70 

1.03 

0.92 

0.71 

0.99 

S.E. 

0.37 

0.31 

0.22 

0.35 

0.35 

0.30 

0.37 

0.22 

0.33 

0.29 

0.22 

0.31 

MAX 

5 

5 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

MIN 

1 

2 

2 

1 

1 

1 

1 

2 

1 

1 

2 

1 

HRC 

n 

■ 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

8 

MEAN 

3.00 

3.40 

3.70 

2.90 

3.50 

2.40 

2.80 

3.70 

3 . 00 

3.50 

3.90 

3.40 

S.D. 

1.15 

0.97 

0.48 

0.99 

0.97 

0.97 

0.79 

0.82 

1.15 

0.85 

0.88 

1.17 

S.E. 

0.37 

0.31 

0.15 

0.31 

0.31 

0.31 

0.25 

0.26 

0.37 

0.27 

0.28 

0.37 

MAX 

4 

5 

4 

4 

5 

4 

4 

5 

4 

4 

5 

5 

MIN 

1 

2 

3 

1 

2 

1 

1 

2 

1 

2 

2 

1 

HRC 

n 

mm 

P 

q 

r 

s 

t 

U 

V 

w 

X 

y 

9 

MEAN 

2.90 

3.00 

3.10 

2.30 

3.20 

2.40 

2.80 

4.00 

3.00 

3.10 

3.90 

3.70 

S.D. 

1.10 

0.67 

0.74 

0.95 

0.79 

1.35 

0.79 

0.94 

1.05 

0.88 

0.57 

0.82 

S.E. 

0.35 

0.21 

0.23 

0.30 

0.25 

0.43 

0.25 

0.30 

0.33 

0.28 

0.18 

0.26 

MAX 

4 

4 

4 

4 

4 

5 

4 

5 

5 

4 

5 

5 

MIN 

1 

2 

2 

1 

2 

1 

2 

2 

2 

2 

3 

3 

HRC 

n 

mm 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

10 

MEAN 

3.70 

3.67 

3.90 

3.50 

4.20 

2.70 

3.10 

4.90 

4.30 

4.10 

4.00 

4.20 

S.D. 

0.67 

0.50 

0.74 

0.71 

0.63 

0.95 

0.74 

0.32 

0.67 

0.32 

0.47 

0.63 

S.E. 

0.21 

0.17 

0.23 

0.22 

0.20 

0.30 

0.23 

0.10 

0.21 

0.10 

0.15 

0.20 

MAX 

5 

4 

5 

5 

5 

4 

4 

5 

5 

5 

5 

5 

MIN 

3 

3 

3 

3 

3 

1 

2 

4 

3 

4 

3 

3 

2 


22 


TABLE  2-5  con't 


HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

11 

MEAN 

1.80 

1.90 

1.70 

1.40 

1.60 

1.10 

1.50 

2.70 

1.70 

2.00 

1.60 

1.80 

S.D. 

0.79 

0.88 

0.48 

0.52 

0.84 

0.32 

0.97 

0.82 

0.82 

0.67 

0.97 

1.23 

S.E. 

0.25 

0.28 

0.15 

0.16 

0.27 

0.10 

0.31 

0.26 

0.26 

0.21 

0.31 

0.39 

MAX 

3 

3 

2 

2 

3 

2 

4 

4 

3 

3 

4 

4 

MIN 

1 

1 

1 

1 

1 

1 

1 

2 

1 

1 

1 

1 

HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

12 

MEAN 

1.50 

1.70 

2.40 

1.70 

1.40 

1.50 

1.20 

2.20 

1.20 

2.00 

2.40 

1.10 

S.D. 

0.71 

0.67 

0.84 

0.95 

0.97 

1.27 

0.63 

0.92 

0.63 

0.82 

0.84 

0.32 

S.E. 

0.22 

0.21 

0.27 

0.30 

0.31 

0.40 

0.20 

0.29 

0.20 

0.26 

0.27 

0.10 

MAX 

3 

3 

4 

3 

4 

5 

3 

4 

3 

4 

4 

2 

MIN 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

HRC 

n 

o 

P 

q 

r 

S 

t 

u 

V 

w 

X 

y 

13 

MEAN 

2.10 

2.10 

2.10 

1.30 

1.50 

1.40 

1.60 

1.90 

1.30 

2.00 

2.10 

2.10 

S.D. 

0.88 

1.10 

0.88 

0.48 

0.71 

0.70 

0.52 

0.88 

0.48 

1.05 

0.74 

0.74 

S.E. 

0.28 

0.35 

0.28 

0.15 

0.22 

0.22 

0.16 

0.28 

0.15 

0.33 

0.23 

0.23 

MAX 

3 

4 

3 

2 

3 

3 

2 

3 

2 

4 

3 

3 

MIN 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

HRC 

n 

O 

P 

q 

r 

s 

t 

U 

V 

W 

X 

y 

14 

MEAN 

2.70 

2.30 

2.30 

1.60 

2.80 

1.10 

2.20 

1.80 

1.80 

2.70 

2.10 

2.60 

S.D. 

0.48 

0.67 

0.67 

1.07 

0.79 

0.32 

1.23 

1.03 

0.79 

0.82 

0.99 

0.84 

S.E. 

0.15 

0.21 

0.21 

0.34 

0.25 

0.10 

0.39 

0.33 

0.25 

0.26 

0.31 

0.27 

MAX 

3 

3 

3 

4 

4 

2 

5 

4 

3 

4 

4 

4 

MIN 

2 

1 

1 

1 

2 

1 

1 

1 

1 

2 

1 

2 

HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

15 

MEAN 

1.55 

1.90 

2.00 

1.40 

1.35 

1.40 

1.75 

2.55 

1.50 

2.15 

1.55 

1.85 

S.D. 

0.76 

0.97 

0.65 

0.60 

0.49 

0.60 

0.85 

0.69 

0.76 

0.88 

0.69 

0.93 

S.E. 

0.17 

0.22 

0.15 

0.13 

0.11 

0.13 

0.19 

0.15 

0.17 

0.20 

0.15 

0.21 

MAX 

3 

4 

3 

3 

2 

3 

3 

4 

3 

4 

3 

4 

MIN 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 


23 


TABLE  2-5  con't. 


HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

16 

MEAN 

1.50 

1.30 

2.40 

1.30 

1.60 

1.30 

2.00 

3.00 

1.90 

2.70 

2.00 

1.50 

S.D. 

0.97 

0.48 

0.70 

0.67 

0.70 

0.48 

1.25 

0.94 

0.88 

0.67 

0.94 

0.97 

S.E. 

0.31 

0.15 

0.22 

0.21 

0.22 

0.15 

0.39 

0.30 

0.28 

0.21 

0.30 

0.31 

MAX 

4 

2 

4 

3 

3 

2 

4 

5 

4 

4 

4 

4 

MIN 

1 

1 

2 

1 

1 

1 

1 

2 

1 

2 

1 

1 

HRC 

n 

■ 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

17 

MEAN 

1.50 

1.55 

2.90 

1.45 

1.60 

1.85 

2.30 

2.90 

1.90 

2.85 

2.75 

1.60 

S.D. 

0.61 

0.69 

0.91 

0.89 

0.60 

0.99 

1.13 

0.72 

0.91 

0.67 

1.02 

0.75 

S.E. 

0.14 

0.15 

0.20 

0.20 

0.13 

0.22 

0.25 

0.16 

0.20 

0.15 

0.23 

0.17 

MAX 

3 

3 

4 

4 

3 

5 

5 

4 

4 

4 

5 

4 

MIN 

1 

1 

1 

1 

1 

1 

1 

2 

1 

2 

1 

1 

HRC 

n 

Dli 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

18 

MEAN 

1.70 

1.90 

2.60 

1.70 

1.90 

1.40 

1.80 

3.30 

1.30 

2.90 

3.10 

1.70 

S.D. 

0.67 

0.57 

0.84 

0.82 

0.74 

0.70 

0.63 

0.67 

0.67 

0.74 

0.57 

0.82 

S.E. 

0.21 

0.18 

0.27 

0.26 

0.23 

0.22 

0.20 

0.21 

0.21 

0.23 

0.18 

0.26 

MAX 

3 

3 

4 

3 

3 

3 

3 

4 

3 

4 

4 

3 

MIN 

1 

1 

1 

1 

1 

1 

1 

2 

1 

2 

2 

1 

HRC 

n 

Dli 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

19 

MEAN 

2.60 

2.50 

3.10 

1.90 

2.00 

1.80 

2.10 

3.70 

2.20 

2.60 

3.20 

3.00 

S.D. 

1.07 

0.97 

0.74 

0.88 

0.82 

0.63 

0.74 

1.06 

0.79 

0.70 

0.92 

0.94 

S.E. 

0.34 

0.31 

0.23 

0.28 

0.26 

0.20 

0.23 

0.33 

0.25 

0.22 

0.29 

0.30 

MAX 

4 

4 

4 

3 

3 

3 

3 

5 

3 

4 

4 

4 

MIN 

1 

1 

2 

1 

1 

1 

1 

1 

1 

2 

1 

1 

HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

20 

MEAN 

2.27 

3.07 

3.63 

2.47 

3.13 

2.40 

3.03 

4.03 

3.00 

3 . 63 

3.60 

3.13 

S.D. 

0.87 

0.87 

0.81 

0.97 

0.82 

0.89 

0.89 

0.76 

0.79 

0.67 

0.86 

1.01 

S.E. 

0.16 

0.16 

0.15 

0.18 

0.15 

0.16 

0.16 

0.14 

0.14 

0.12 

0.16 

0.18 

MAX 

4 

5 

5 

4 

4 

4 

4 

5 

4 

5 

5 

5 

MIN 

1 

1 

1 

1 

1 

1 

1 

2 

1 

2 

1 

1 

2 


24 


TABLE  2-5  con't 


HRC 

n 

o 

P 

q 

r 

s 

t 

u' 

V 

w 

X 

y 

21 

MEAN 

2.50 

2.00 

3.10 

1.90 

2.30 

2.20 

2.90 

3.50 

2.40 

3.30 

3.10 

2.50 

S.D. 

0.71 

0.82 

0.74 

0.57 

0.82 

1.14 

0.32 

0.53 

0.84 

0.67 

0.74 

0.85 

S.E. 

0.22 

0.26 

0.23 

0.18 

0.26 

0.36 

0.10 

0.17 

0.27 

0.21 

0.23 

0.27 

MAX 

4 

3 

4 

3 

3 

4 

3 

4 

4 

4 

4 

4 

MIN 

2 

1 

2 

1 

1 

1 

2 

3 

1 

2 

2 

1 

HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

22 

MEAN 

2.80 

3.50 

4.10 

3.30 

3.80 

2.00 

2.90 

4.20 

4.00 

3.70 

3.50 

4.00 

S.D. 

0.79 

0.97 

0.88 

0.67 

0.63 

0.82 

0.74 

0.92 

0.82 

1.06 

0.85 

1.25 

S.E. 

0.25 

0.31 

0.28 

0.21 

0.20 

0.26 

0.23 

0.29 

0.26 

0.33 

0.27 

0.39 

MAX 

4 

5 

5 

4 

4 

3 

4 

5 

5 

5 

4 

5 

MIN 

1 

2 

3 

2 

2 

1 

2 

2 

2 

1 

2 

1 

HRC 

n 

o 

P 

q 

r 

s 

t 

u 

V 

w 

X 

y 

23 

MEAN 

2.10 

3.70 

3.80 

3.40 

4.20 

2.70 

3.50 

3.90 

3.50 

3.90 

3.60 

2.60 

S.D. 

0.88 

0.48 

1.03 

0.70 

0.92 

1.16 

0.71 

0.57 

0.53 

1.10 

0.70 

0.97 

S.E. 

0.28 

0.15 

0.33 

0.22 

0.29 

0.37 

0.22 

0.18 

0.17 

0.35 

0.22 

0.31 

MAX 

4 

4 

5 

4 

5 

4 

4 

5 

4 

5 

4 

4 

MIN 

1 

3 

2 

2 

2 

1 

2 

3 

3 

1 

2 

1 

HRC 

n 

o 

P 

q 

r 

s 

t 

U 

V 

w 

X 

y 

24 

MEAN 

3.60 

3.56 

4.00 

2.90 

3.40 

3.10 

3.90 

4.40 

3.90 

3.90 

3.60 

3.20 

S.D. 

0.97 

1.13 

1.25 

0.99 

0.84 

0.99 

0.57 

0.97 

1.10 

0.88 

0.84 

1.14 

S.E. 

0.31 

0.38 

0.39 

0.31 

0.27 

0.31 

0.18 

0.31 

0.35 

0.28 

0.27 

0.36 

MAX 

5 

5 

5 

4 

4 

4 

5 

5 

5 

5 

4 

5 

MIN 

2 

2 

2 

1 

2 

1 

3 

2 

2 

2 

2 

1 

HRC 

n 

■ 

P 

q 

r 

s 

t 

U 

V 

w 

X 

y 

25 

MEAN 

3.70 

3.20 

3.80 

3.40 

3.90 

2.50 

3.30 

4.30 

4.00 

3.80 

4.20 

4.40 

S.D. 

0.48 

0.79 

0.92 

0.52 

0.57 

0.85 

0.67 

0.48 

0.67 

0.42 

0.63 

0.70 

S.E. 

0.15 

0.25 

0.29 

0.16 

0.18 

0.27 

0.21 

0.15 

0.21 

0.13 

0.20 

0.22 

MAX 

4 

4 

5 

4 

5 

4 

4 

5 

5 

4 

5 

5 

MIN 

3 

2 

2 

3 

3 

1 

2 

4 

3 

3 

3 

3 
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3.0  OBJECTIVE  TESTING 


3.1  Executive  Summary 

The  Objective  Testing  reported  herein  is  required  by  T1A1.5  to  support 
development  of  transmission  standards  for  Video  Telephony  and  for  Video 
Teleconferencing. 

Results  of  the  Objective  Tests  have  resulted  in  Fair  to  Poor  correlation  with 
the  Subjective  Testing. 

3.2  Test  Location 

All  objective  tests  done  by  Delta  Information  Systems  were  performed  in  the 
Engineering  Laboratory  in  their  facility  at  300  Welsh  Road,  Horsham,  Pennsylvania. 

3.3  Test  Performed 

Delta  has  designed  a  series  of  computer-generated  test  patterns  for  objective 
motion  tests.  They  make  it  possible  to  define  and  numerically  measure  certain 
motion  parameters.  The  two  types  of  test  patterns  are  the  rotating  wheel  and 
switched  dots.  The  rotating  wheel  patterns  feature  three  different  spoke  widths, 
which  combined  with  six  to  nine  rotation  speeds,  produce  23  patterns.  These 
patterns  are  included  in  the  betacam  tapes  along  with  the  25  scenes  processed  by 
25  HRCs  and  are  analyzed  by  measurement  of  temporal  response  and  transmitted 
frame  rate.  The  switched  dot  patterns  feature  three  different  white  dots 
alternating  with  a  black  background.  Analysis  of  the  dot  pattern  is  by  observing 
the  frames  required  for  full  amplitude  response  following  a  pattern  switch  (scene 
cut  response  {SCR}). 

It  was  decided  that  the  emphasis  for  this  year  would  be  placed  on  SCR 
measurements. 
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3.3.1  Test  Procedure 


a.  Using  the  large  dot  pattern,  freeze  the  first  frame  following  the  scene  cut. 

b.  Select  a  line  near  the  top  of  the  scene  which  passes  through  the  center  of  a 
selected  white  dot.  Record  the  dot  amplitude. 

c.  Repeat  this  measurement  on  at  least  two  other  lines  near  the  center  and  at 
the  bottom  of  the  scene.  Average  the  results. 

d.  Advance  the  tape  frame,  and  record  all  changes  in  amplitude  until  the  dot 
white  amplitude  approximates  100%. 

e.  The  number  of  frames  required  to  achieve  full  brightness  is  a  measure  of  the 
image  update  time. 

f.  Repeat  for  medium  and  small  dot  patterns. 

3.4  Test  Result 


The  SCR  for  several  HRCs  are  shown  in  Table  3-1.  The  measurement  made 
is  of  the  number  of  frames  required  for  the  peak  white  of  the  dot  pattern  to 
achieve  the  steady-state  level.  These  measurements  were  made  for  the  large, 
medium,  and  small  dot  patterns.  It  is  noted  that  some  HRCs  are  quicker  for  the 
small  dots  while  some  respond  more  slowly  for  small  dots.  It  is  also  noted  that 
several  HRCs  show  no  measurable  delay  for  all  three  dot  patterns.  The  analysis  of 
these  results  have  concluded  that  the  test  conducted  on  the  recorded  dot  patterns 
needs  a  time  code  for  monitoring  signal  transient  response  measurements. 
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TABLE  3-1  -  SUMMARY  OF  TEST  RESULTS 


NO. 

HRC 

SCENE  CUT  RESPONSE 

(Number  of  TV  Frames) 

ALGORITHM 

DATA 

RATE 

CODER/ 

DECODER 

LARGE 

DOTS 

MEDIUM 

DOTS 

SMALL 

DOTS 

■ 

PROPRIETARY 

VECT. 

QUANTI¬ 

ZATION 

128 

- 

9 

12 

21 

5 

384 

• 

3 

4 

7 

6 

PROPRIETARY 

TYPE  C 

128 

3 

2 

1 

■ 

384 

- 

2 

1 

2 

8 

768 

- 

1 

1 

1 

9 

PROPRIETARY 

TYPE  D 

768 

- 

2 

4 

7 

10 

1536 

- 

2 

3 

4 

11 

H.261-QCIF 

128 

DIFF 

2 

2 

2 

12 

H.261-QCIF 

128 

SAME 

2 

1 

1 

13 

H.261-QCIF 

168 

SAME 

0 

0 

0 

14 

H.261-QCIF 

384 

DIFF 

2 

2 

2 

15 

H.261-CIF 

128 

SAME 

0 

0 

0 

16 

H.261-CIF 

128 

SAME 

0 

0 

0 

17 

H.261-CIF 

128 

DIFF 

0 

0 

0 

19 

H.261-CIF 

256 

SAME 

0 

0 

0 
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4.0  COMPARISON  OF  SUBJECTIVE  AND  OBJECTIVE  TESTS 


The  results  of  both  the  subjective  and  objective  tests  are  shown  in  Table 
4-1 .  Shown  in  this  Table  are  the  Mean  Subjective  Score  for  three  groupings,  as 
follows: 

a.  Overall  -  All  scenes  except  h,  i,  and  y; 

b.  VT  (Video  Telephony)  -  scenes  f,  j,  k,  and  I; 

c.  VTC  (Video  Teleconferencing)  -  scenes  d,  g,  o,  p,  q,  and  r. 

A  review  of  the  results  of  subjective  and  objective  measurements  show 
significant  inconsistency.  Why  would  the  SCR  measure  so  good  (quick  response) 
when  the  subjective  performance  is  poor?  The  answer  appears  to  be  that  video 
encoders  employ  widely  different  coding  strategies  when  a  sudden  scene  change 
occurs,  such  as  a  scene  cut.  One  manufacturer  may  feel  that  the  user  prefers  to 
see  the  scene  built  up  dynamically  from  the  original  image  to  the  new  image.  HRC 
4  is  an  example  of  such  a  case  where  the  bit  rate  is  low  (128  kb/s),  the  quality  is 
low  (2.35),  and  the  SCR  is  correspondingly  poor  (21  frames  for  small  dots). 
Contrarily,  other  vendors  (see  HRC  1 1  to  19)  may  feel  that  the  user  would  be 
distracted  by  the  dynamically  changing,  distorted  transitional  images  and  would 
prefer  to  take  a  long  time  to  transmit  the  first  new  frame  with  very  high  quality.  In 
such  a  case,  the  response  to  the  scene  cut  takes  the  form  of  a  delayed,  accurate 
reproduction  rather  than  a  slow  buildup  of  many  distorted  frames.  In  both  cases, 
the  input  scene  cut  is  distorted;  it  is  merely  a  matter  of  the  type  of  distortion 
which  different  vendors  consider  least  disturbing  to  the  eye. 

It  is  possible  to  develop  a  more  rigorous  test  procedure  for  measuring  the 
delay  in  the  scene  cut  response  in  the  case  of  HRCs  1 1  to  19,  or  the  transition 
time  in  the  case  of  HRCs  4,  5,  9,  and  10.  Unfortunately,  the  video  tapes 
containing  the  video  test  patterns  used  for  SCR  did  not  have  time  coding. 
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TABLE  4-1  -  SUMMARY  OF  TEST  RESULTS 


NO. 

HRC 

MEAN  SUBJ.  SCORE 

SCENE  CUT  RESPONSE 
(Number  of  TV  Frames) 

ALGORITHM 

DATA 

RATE 

CODER/ 

DECODER 

OVERALL 

VT 

VTC 

LARGE 

DOTS 

MEDIUM 

DOTS 

SMALL 

DOTS 

B 

PROPRIETARY 

VECT. 

QUANTI¬ 

ZATION 

128 

- 

2.35 

3.26 

2.25 

9 

12 

21 

5 

384 

- 

3.21 

3.98 

3.07 

3 

4 

7 

PROPRIETARY 

TYPE  C 

128 

- 

2.62 

3.58 

2.33 

3 

2 

1 

■ 

384 

- 

3.18 

3.65 

3.22 

2 

1 

2 

8 

768 

• 

3.47 

4.00 

3.48 

1 

1 

1 

9 

PROPRIETARY 

TYPE  D 

768 

- 

3.39 

4.18 

3.17 

2 

4 

7 

10 

1536 

- 

4.22 

4.73 

3.89 

2 

3 

4 

11 

H.261-QCIF 

128 

DIFF 

2.00 

2.40 

1.83 

2 

2 

2 

12 

H.26VQCIF 

128 

SAME 

1.95 

2.60 

1.93 

2 

1 

1 

13 

H.261-QCIF 

168 

SAME 

1.97 

2.65 

1.90 

0 

0 

0 

14 

H.261-QCIF 

384 

DIFF 

2.37 

3.10 

2.22 

2 

2 

2 

15 

H.261-CIF 

128 

SAME 

2.00 

2.63 

1.81 

0 

0 

0 

16 

H.261-CIF 

128 

SAME 

2.23 

2.75 

1.93 

0 

! 

0 

0 

17 

H.261-CIF 

128 

DIFF 

2.47 

3.08 

2.19 

0 

0 

0 

19 

H.261-CIF 

256 

SAME 

2.75 

3.38 

2.63 

0 

0 

0 

5.0  CONCLUSIONS  AND  RECOMMENDATIONS 


5.1  Conclusions 

The  rather  poor  correlation  of  the  Objective  SCR  with  Subjective  scoring  of 
video  scenes  is  attributed  to  changes  in  the  encoding  strategies  on  the  assumption 
that  users  may  prefer  not  to  see  the  picture  dynamically  change  when  the  input 
scene  changes  radically.  Consequently,  the  SCR  may  need  to  yield  to  another 
objective  test  method  for  predicting  the  subjective  performance  of  a  video  system. 

5.2  Recommendations 

Although  the  results  of  the  SCR  were  disappointing,  a  great  deal  has  been 
learned  about  the  problem  of  predicting  system  performance.  It  is  very  possible 
that  the  experience  gained  can  be  used  to  identify  new  objective  test  procedures 
which  show  promise  of  success. 

Two  examples  of  work  which  might  be  done  are  listed  below. 

Rotating  Wheel  Tests  -  When  the  Work  Plan  for  1994  was  prepared,  work  was 
focused  on  the  SCR  rather  than  the  Rotating  Wheel  (RW)  objective  test  since 
recent  tests  had  indicated  more  promise  for  the  SCR  and  there  was  limited 
resources  to  investigate  both.  Although  some  of  the  recent  tests  on  the  RW  were 
not  highly  successful,  it  was  not  concluded  that  the  RW  could  not  succeed  to 
provide  a  useful  objective  test.  Therefore,  it  is  proposed  to  re-examine  the  rotating 
wheel  test  pattern,  with  the  benefit  of  the  SCR  experience,  to  determine  whether 
there  is  an  effective  way  to  use  it  for  an  objective  test.  If  successful,  this 
approach  has  the  advantage  that  the  RW  approach  can  be  tested  for  validity  using 
existing  T1A1.5  video  tapes. 

Scene  Cut  Response  Tests  -  It  may  be  possible  to  develop  an  SCR  test  which  is 
sufficiently  robust  that  it  would  be  able  to  yield  a  single  universal  measure  of 
distortion  to  an  input  scene  cut  regardless  whether  the  codec  created  a  transition 
or  delay  distortion. 
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SAMPLE  IMAGES  OF  TEST  SCENES 


1.  INTRODUCTION 

This  contribution  provides  a  angle  image  from  each  of  the  25  video  test  scenes  that  were  used  in 
the  T1A1.5  VTcAnr  subjective  and  objective  testing.  The  following  table  gives  the  scene  letter 
designation  (a-y),  the  six-letter  processing  name,  a  short  description,  and  the  spatial  information 
(SI)  and  temporal  infonnation  (TI)  values  calculated  as  documented  in  the  Appendix  to  the  Draft 
standaiti  TlAl.5/94-107. 

Table  1: 


Scene 

Letter 

Scene 

Name 

Description 

SI 

TI 

a 

vtc2mp 

Woman  at  map,  talking 

82.3 

4.3 

b 

vtc2zm 

Woman  at  map  with  pointer,  zoom 

95.0 

22.9 

c 

washdc 

Map  with  hand  &  pencil 

130.5 

17.0 

d 

3inrow 

Man  talking,  pan 

81.7 

30.8 

c 

boblec 

Bob’s  lecture  at  chalkboard 

64.8 

21.9 

f 

vtclnw 

News  Story  (woman) 

56.2 

5.3 

g 

5rowl 

Five  people  in  a  row  1 

77.3 

8.5 

h 

flogar 

Flower  garden  with  windmill,  pan 

227.0 

46.4 

i 

ftball 

Football  game 

156.6 

53.7 

j 

susie 

Susie  on  telephone 

58.7 

24.6 

k 

disguy 

Announcer  (man) 

48.8 

8.8 

1 

disgal 

Miss  America 

34.0 

m 

smityl 

Salesman  1  (with  box) 

152.0 

30.3 

n 

smity2 

Salesman  2  (with  magazine) 

154.5 

35.1 

0 

intros 

Introductions,  with  pans 

78.1 

41.6 

p 

3twos 

Three  pairs  of  people,  cuts 

63.0 

67.2 

q 

2wbonl 

Two  people  at  whiteboard 

68.7 

66.8 

r 

splitti 

Split  screen,  6  people 

92.5 

11.6 

s 

cirkit 

Circuit  diagram,  zoom 

237.7 

48.0 

t 

rodmap 

Roadmap,  with  pen 

170.4 

42.6 

u 

filter 

Diagram  on  yellow  pad 

55.5 

15.4 

V 

ysmite 

Topographic  map  of  Yosemite 

88.7 

16.0 

w 

vowels 

Woman  at  whiteboard 

114.3 

17.8 

X 

inspec 

Woman  with  viewgraph 

46.8 

12.8 

y 

fredas 

Fred  Astaire  dancing 

6l0 

23.8 
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Figure  1.  T1A1.5  VTCAHr  Test  Scenes — Sample  Images 


a)  vtc2mp 
c)  washdc 


b)  vtc2zm 
d)  Sinrow 


Figure  5.  T1A1.5  VTC/VT  Te 

u)  filter 
w)  vowels 


y) 


Scenes — Sample  Images 


v)  ysmite 
x)  inspec 
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ABSTRACT: 

This  is  the  subjective  test  plan  required  by  Tl  A1.5  to  complete  its  video  performance  specification  project. 
The  Detailed  Test  Plan  Ad  Hoc  Group  prepared  this  plan  for  the  February  8,  1993  meeting,  where  version 
TlAl. 5/93-014  Rl  was  originally  accepted  by  the  full  Working  Group. 

In  the  process  of  developing  the  Data  Analysis  Plan,  some  of  the  calculations  originally  envisioned  were 
deemed  unnecessary.  The  Data  Analysis  Ad  Hoc  Group  was  then  free  to  develop  a  more  balanced 
sampling  plan  and  make  other  improvements  to  the  subjective  test  plan. 

Revisions  and  new  material  from  subsequent  meetings  of  the  Data  Analysis  Ad  Hoc  Group  were 
adopted  as  version  TlAl.5/93-014  R5  (on  a  technical  basis)  at  the  August  9,  1993  Working  Group 
meeting.  Some  changes  and  additional  appendices  were  adopted  at  the  November  1993,  January 
1994,  and  July  1994  meetings  of  the  Working  Group, 
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Video  Performance  Standard  Subjective  Test  Plan 


1.  INTRODUCTION 

Working  Group  T1A1.5  is  in  the  process  of  preparing  an  ANSI  standard  on  video 
teleconference  system  performance  measurement.  The  process  includes  steps  to  identify 
objective  measures  of  video  performance,  to  compare  the  objective  measures  with  user 
opinion  of  video  quality,  and  to  select  from  the  candidate  measures  those  that  are  well- 
correlated  with  user  opinion,  as  only  these  measures  offer  the  desired  information. 

During  the  T1A1.5  meetings  in  October,  1992,  agreement  was  reached  on  a  set  of  25 
video  test  scenes  and  on  25  Hypothetical  Reference  Circuits  (HRC).  The  test  scenes, 
along  with  candidate  objective  test  waveforms,  have  been  assembled  on  D2  format  video 
tape  and  played  through  the  25  HRC’s.  The  25  by  25  matrix  results  in  625  different  test 
combinations,  where  a  test  combination  is  defined  as  the  record  of  a  single  scene 
transmitted  through  a  single  HRC. 

To  correlate  the  candidate  measures  with  a  representative  user’s  view  of  video  quality,  an 
estimate  of  the  perceived  quality  of  each  test  combination  must  be  available.  Working 
Group  T1A1.5  formed  an  Ad  Hoc  Group  to  develop  a  detailed  procedure  for  both 
subjective  and  objective  testing.  This  document  describes  the  subjective  test  procedure 
to  be  followed  at  each  of  the  three  test  laboratories  (Delta  Information  Systems,  GTE 
Labs,  and  NTIA-ITS),  in  which  the  above  test  combinations  become  the  stimuli  for  a 
video  grading  task. 

The  procedure  will  follow  CCIR  Recommendation  500-5  in  general.  This  document 
identifies  the  specific  sections  and  procedures  to  use,  since  there  are  many  options  within 
the  Recommendation.  Further,  additional  details  specific  to  this  test  program  will  be 
defined  here,  such  as  the  number  of  subjects  required  to  view  each  test  combination,  and 
the  division  of  test  combinations  between  the  three  labs.  In  this  way,  the  controlled 
conditions  and  test  delivery  will  be  determined  and  maximal  consistency  among  the 
laboratories  should  result. 

The  GOAL  of  this  Test  Procedure  is  to  estimate  the  distribution  of  opinion  of  video 
teleconference  system  users  when  presented  with  representative  video  sequences. 

The  following  points  represent  the  group’s  philosophy  concerning  the  standard’s 
development: 

1.  The  intent  of  this  process  is  to  study  the  relative  performance  of  a  set  of  proposed 
objective  video  measures  as  predictors  of  subjective  Judgement. 

2.  The  process  is  an  evolutionary  one,  and  the  membership  does  not  necessarily 
expect  to  reach  the  final  set  of  measures  in  a  single  step,  or  single  cycle  through  the 
process. 

3.  While  negative  contingencies  or  failures  of  the  process  may  not  be  defined  to  the 
last  detail,  the  membership  believes  that  they  have  experience  necessary  to 
recognize  when  the  results  indicate  failure.  Further,  that  they  are  willing  to 
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proceed  with  this  research  process  with  an  understanding  of  the  risks. 

4.  There  is  a  need,  raised  here  and  previously  by  other  members,  to  develop  a 
document  which  describes  the  data  analysis  process  step  in  some  detail.  This 
document  should  exist  prior  to  the  completion  of  the  testing  steps.  (The  completed 
data  analysis  plan  appears  in  Section  5.) 

5.  It  is  recognized  that  the  precision  of  the  conclusions  drawn  as  to  the  relationship 
between  objective  and  subjective  performance  will  be  based  on  the  precision  of  the 
basic  subjective  testing.  The  Working  Group  in  this  plan  will  assess  the  level  of 
precision  needed  for  the  results  to  be  compelling  and  to  receive  industry  consensus. 

In  order  to  be  useful  in  the  standard  development  process,  this  procedure  is  also 
consistent  with  the  Scope  and  Purpose  of  the  Draft  VTC  Performance  Standard 
(TlAl.5/94-107). 

2.  MAJOR  TEST  DESIGN  ATTRIBUTES 

This  section  contains  the  consensus  position  of  the  Ad  Hoc  Group  on  several  major  areas 
that  required  determination  before  any  subjective  tests  could  begin. 

The  general  starting  point  was  this  list  of  design  requirements: 

1 .  Test  a  broad  range  of  Hypothetical  Reference  Circuit  types. 

2.  Use  a  broad  range  of  Test  Scenes. 

3.  Recruit  an  adequate  number  of  viewers  representing  a  well-defined  target 
population. 

4.  Test  as  many  of  these  combinations  as  feasible. 

5.  Adopt  a  partially  balanced  design  which  ensures  that  the  quantities  of  interest  are 
not  confounded  with  unmeasurable  sources  of  variation. 

6.  Use  equal-probability  sampling  if  possible. 

7.  Include  appropriate  quality  checks. 

8.  Use  CCIR  Recommendation  500-5  as  a  guideline. 

9.  Use  Digital  Play/Record  and  editing  to  minimize  generation  loss. 

10,  Test  the  subjects  for  suitability  (i.e.  vision  acuity). 

2.1  Test  Matrix 

As  stated  earlier,  agreement  was  reached  on  a  set  of  25  video  test  scenes  and  on  25 
Hypothetical  Reference  Circuits  (HRC).  The  25  by  25  matrix  results  in  625  different  test 
combinations,  as  shown  in  Table  1. 
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Table  1.  TEST  MATRIX 


SCENES 

HYPOTHETICAL  REF,  CIRCUITS 

1 

2 

3 

4 

. 

. 

. 

25 

a 

al 

a2 

a3 

... 

a25 

b 

bl 

b2 

... 

c 

cl 

•••  . 

d 

... 

. 

• 

y 

yi 

... 

y25 

2.2  Number  of  Test  Subjects 

The  objective  of  these  tests  is  to  obtain  an  experimental  Mean  Opinion  Score  (MOS)  for 
each  test  combination  where  the  value  obtained  differs  by  no  more  than  ±  0.2  opinion 
score  points  from  the  true  mean  value  with  95%  confidence.  30  lest  subjects  are 
estimated  as  required.  The  sample  size  was  determined  using  the  following  method. 

First,  the  standard  deviation,  s,  was  estimated  (for  this  experiment)  by  review  of  previous 
experimental  results.  Since  the  range  of  s  was  found  to  be  large  (0. 1  <5  <  1.0  on  a  five 
point  grading  task),  a  representative  value  for  s  was  chosen.  It  is 

j  =  0.5 

Using  the  following  equation,  we  determined  the  necessary  sample  size  to  meet  the  0.2 
score  point  confidence  interval  requirement,  e. 

qtiO.975,  n)  X  s 

where  qt(0.915,  n)  is  a  percentile  of  the  Student’s  t  distribution  for  double-sided 
confidence  intervals  at  95%  and  n  is  the  unknown  sample  size. 

When  n  =  30,  =0. 186  and  the  requirement  is  satisfied.  For  the  test  combinations  where 
the  sample  standard  deviation  is  more  than  0.5,  slightly  larger  confidence  intervals  will 
result. 

Test  labs  must  provide  30  viewer  opinions  for  each  test  combination  as  a  minimum  after 
screening  (see  Section  2.6).  Viewers  are  expected  to  rate  all  test  combinations  that  are 
shown  to  them.  The  entire  data  set  for  the  30  viewers  must  be  provided. 

2.3  Target  Population  and  Viewer  Qualifications 

Viewers  selected  for  this  experiment  must  have  normal  or  corrected-to-normal  visual 
acuity  and  color  vision.  These  faculties  will  be  tested  prior  to  participation.  Viewers  that 
do  not  meet  these  requirements  must  not  be  included  in  the  sample. 

In  keeping  with  the  goal  of  the  plan,  the  viewers  should  represent  typical  video 
telephony/teleconference  (VTA^TC)  system  users.  Ideally,  they  should  be  persons  who 
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use  these  systems  now,  or  envision  using  them  in  the  next  few  years.  The  viewers  should 
not  be  persons  working  directly  on,  or  in  support  of,  the  design,  sales,  maintenance,  or 
performance  assessment  of  VTA^TC  systems  or  services. 

There  is  one  additional  qualification.  50%  of  the  viewers  at  each  lab  should  have  some 
experience  with  video  conferencing.  Some  allowances  may  be  made  for  recruitin'^ 
difficulties.  ° 

2.4  Stimuli  Presentation  and  Voting  Method 

CCIR  Rec.  500-5  describes  several  test  methodologies  for  subjective  assessment  of 
television  pictures.  The  grading  scale  used  determines  the  user’s  measure  of  scene 
rendition.  The  ANSI  standard  for  expressing  video  performance  will  make  reference  to 
this  scale. 

The  Working  Group  agreed  on  the  double-stimulus/impairment  scale  method  as 
described  in  section  2  of  CCIR  500-5,  with  some  modifications.  The  modifications  are: 

1.  The  rating  scale  used  by  subjects  will  not  show  the  numerical  values  5  through  1. 
These  values  (5  =  Imperceptible)  will  be  assigned  during  data  entry. 

2.  The  reduction  of  time  intervals  as  defined  in  the  presentation  of  test  material 
(Figure  2  of  Rec.  500-5),  to  allow  9  seconds  to  view  the  reference  scene,  a  3 
second  gap,  9  seconds  to  view  the  impaired  scene,  and  a  9  second  voting  interval. 

3.  A  mid-gray  level  of  50  IRE  will  be  used  in  the  interval  between  pairs  of  scenes  and 
during  the  voting  interval. 

2.5  Voting  Forms 

Appendix  B  contains  a  sample  voting  form  which  is  the  result  of  collaborative  effort  on 
the  part  of  many  working  group  members. 

2.6  Quality  Checks 

There  are  four  necessary  quality  checks: 

1.  Viewer  reliability  will  be  tested  through  repetition  of  one  test  combination  in  every 
session.  Viewers  will  be  disqualified  if  their  grades  differ  by  more  than  2  opinion 
score  points  (of  the  1  through  5  scale)  for  the  repeated  test  combination  in  any  test 
session.  Combinations  selected  for  repetition  will  come  from  HRC’s  with  either 
384  kbps  or  768  kbps  transmission  speed,  so  as  to  avoid  combinations  whose 
expected  average  rating  scores  are  near  either  end  of  the  rating  scale. 

2.  Viewer  reliability  will  also  be  tested  through  the  distribution  of  Null  HRC 
conditions  among  all  test  labs.  Each  test  session  will  contain  at  least  1  Null 
combination.  Viewers  will  be  disqualified  if  they  grade  the  Null  combination  at  3 
or  less  (of  the  1  through  5  scale)  in  any  test  session. 

3.  As  many  as  2  missing  ratings  will  be  tolerated  per  viewer.  If  any  missing  rating  is 
on  a  quality  check  combination,  then  the  viewer  will  be  disqualified. 
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4.  Lab-to-lab  consistency  will  also  be  tested  through  repetition  of  75  combinations  at 
each  of  the  labs.  See  the  following  section. 

2.7  Partially  Balanced  Sampling  Plan 

2.7.7  Allocation  of  HRCs  and  Scenes  to  Video  Tapes 

Setting  aside  four  HRC’s  for  special  treatment,  21  HRC’s  were  combined  into  a  partially 
balanced  design  described  here. 

Considerations  of  viewer  burden  allows  us  to  show  only  about  one-third  of  the  possible 
test  combinations  to  any  single  viewer,  spread  across  several  sessions.  It  was  therefore 
decided  to  create  three  sets  of  viewing  tapes  (designated  as  the  "Red",  "Green"  and 
"Orange"  sets,  or  R,  G  and  O,  for  short),  each  set  of  tapes  containing  all  the  scenes,  but 
only  one-third  of  the  HRC’s,  in  all  possible  combinations.  Any  given  viewer  will  see 
exactly  one  set  of  tapes. 

Referring  to  the  HRC’s  by  number  in  accordance  with  document  T1 A  1.5/92- 174  (see 
Appendix  D),  we  allocated  the  HRC’s  to  the  sets  of  viewing  tapes  as  follows: 

Red  Tape  Set:  1,  4,  7,  8,13,15,19,20,22,24 

Green  Tape  Set:  2,  5,  6,  10,  14,  15,  16,  17,  20,  23 

Orange  Tape  Set:  3,  4,  9,11,12,17,18,20,21,25 

This  allocation  was  guided  by  a  desire  to  include  the  full  range  of  video  performance  in 
each  tape  set,  observing  that  the  HRC’s  fall  into  9  general  types  according  to  their 
engineering  descriptions  (see  Table  3).  In  particular: 

—  An  even  division  was  achieved  among  codec  types  and  transmission  rates. 

—  Each  tape  set  has  2  or  3  proprietary  HRC’s,  1  or  2  QCIF  HRC’s,  5  CIF  HRC’s  (not 
counting  the  Null  or  VHS  HRC). 

—  Each  tape  set  contains  one  HRC  with  transmission  errors. 

Additionally,  the  four  remaining  HRC’s  that  had  been  set  aside  at  the  start  were  given 
special  treatment  by  being  included  in  more  than  one  tape  set,  each.  The  purpose  is  to 
allow  post-hoc  calibration  checks  between  the  tape  sets.  They  were  allocated  to  the  tape 
sets  as  follows: 


TAPES 

HRC  No. 

Description 

R.G,0 

20 

Identical  Px64  Codecs  at  384  kB/s 

R.G 

15 

Identical  Px64  Codecs  at  1 12  kB/s 

G,0 

17 

Different  Px64  Codecs  at  128  kB/s 

R.O 

4 

Vector  Quantiz.  Codec  at  128  kB/s 

2.7.2  Allocation  of  Tape  Sets  to  Testing  Labs 

Three  different  video  labs  volunteered  to  participate  in  the  subjective  viewing  and  data 
gathering  phase  of  the  study.  Early  thinking  called  for  sending  each  of  the  three  color- 
coded  sets  of  tapes  to  just  one  lab,  which  would  have  allocated  each  HRC  to  just  a  single 
lab  "  except  for  the  four  HRC’s  that  are  repeated  across  2  or  3  labs.  This  plan  had  a 
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certain  appeal  in  terms  of  its  logistic  simplicity,  but  it  has  the  drawback  that  any  inter¬ 
laboratory  differences  would  have  to  be  assessed,  and  possibly  corrected  for,  using  only  a 
tiny  fraction  of  the  test  material  and  data. 

Other  similar  studies  suggest  that  inter-laboratory  differences  might  indeed  occur  -- 
either  because  of  uncontrollable  differences  in  the  physical  conditions  of  the  test  set-up, 
or  because  of  differences  in  the  sampled  populations  of  viewers  at  the  three  locations. 
We  did  not  want  to  be  left  with  a  set  of  data  in  which  quantities  of  primary  interest  (HRC 
quality  ratings)  are  essentially  confounded  with  unmeasurable,  irrelevant,  and  possibly 
inexplicable  factors  (collectively  called  "inter-laboratory  differences"),  merely  because  of 
weaknesses  in  the  sampling  design. 

So  the  initial  plan  was  abandoned  in  favor  of  a  more  balanced  design,  in  which  all  of  the 
color-coded  tape  sets  are  sent  to  each  viewing  lab.  Every  lab  is  instructed  to  divide  its  30 
test  subjects  randomly  into  three  teams  of  10  subjects  each,  for  viewing  the  three  sets  of 
tapes.  Thus,  each  lab  will  assemble  a  "Red  Team",  a  "Green  Team"  and  an  "Orange 
Team".  In  the  overall  data  set,  the  "Red  Cohort"  will  be  the  union  of  the  Red  Teams 
from  the  three  labs,  and  will  be  spread  in  equal  numbers  across  the  labs,  and  so  on  for  the 
other  colors.  Schematically,  we  have: 


Lab 

Tapes 

Viewers 

Alternates 

X 

Red 

XI 

,.X10 

Xll  .. 

X20 

Green 

X21  .. 

,.X30 

X31  .. 

.X40 

Orange 

X41  .. 

,.  X50 

X51  .. 

.X60 

Y 

Red 

Y1  .. 

.  YIO 

Yll  .. 

.  Y20 

Green 

Y21  .. 

.  Y30 

Y31  .. 

.  Y40 

Orange 

Y41 .. 

.  Y50 

Y51  .. 

.  Y60 

Z 

Red 

Z1  .. 

.ZIO 

Zll  .. 

.Z20 

Green 

Z21  .. 

.Z30 

Z31  .. 

.Z40 

Orange 

Z41  .. 

.Z50 

Z51 .. 

.Z60 

(N.B.  the  viewer  sequence  numbers  above  are  their  numbers  after  being  randomized 
according  to  instmctions  in  Section  2.9.  The  numbers  for  alternate  viewers  may  not  be 
completely  used.) 

Since  the  quantities  of  primary  interest  in  the  data  analysis  will  be  summaries  across  the 
cohorts,  any  laboratory-specific  factors  affecting  judgements  will  be  neutralized  by  being 
equally  spread  out  across  all  results. 

This  plan  turns  a  potential  liability  (inter-laboratory  differences)  into  a  strength:  the 
pooled  data  set  from  the  three  labs  can  be  regarded  as  a  properly  stratified  random  sample 
from  a  target  population  that  is  an  equal-probability  mixture  of  the  target  populations 
realized  at  the  three  labs.  In  this  way,  by  going  to  three  labs  instead  of  one  for  test 
subjects,  we  are  likely  to  broaden  the  scope  of  our  sampled  population  and  make  it  more 
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representative  of  the  true  potential  market  for  video  teleconferencing  in  the  country. 

This  plan  has  the  further  desirable  property  that  any  given  HRC  has  the  same  probability 
of  being  viewed  by  every  test  subject  in  the  study,  which  justifies  the  use  of  unweighted 
averages  across  cohorts  as  efficient  and  unbiased  estimates  of  population  parameters. 

Moreover,  this  design  will  also  allow  us  to  study  the  inter-laboratory  differences 
themselves  -  with  a  view  to  clarifying  our  understanding  of  the  target  sub-population 
that  were  actually  sampled. 

2.8  Stimuli  Presentation  Order 

2.8. 1  Number  of  Test  Sessions 

It  was  observed  that  with  some  slight  modifications  to  the  test  method  outlined  in  CCIR 
Rec.  500-5,  the  required  number  of  stimuli  that  we  must  present  to  each  viewer  can  be 
accommodated  in  4  viewing  sessions  of  32.5  minutes  each  —  exceeding  the  CCIR 
recommendation  by  just  2.5  minutes.  Each  session  will  consist  of 

10  HRC’s  *  25  Scenes  *  0.5  min/seq.  =  125  min  of  testing 
125  min  /4  sess  =  31.25  min/sess 


To  each  session,  we  add  1  minute  for  two  additional  calibration  checks  (one  repeated 
combination  and  one  Null  combination).  This  gives  an  average  of  32.25  min  per  session, 
or  two  sessions  with  32.5  minutes  and  two  with  32.0  minutes. 

2.8.2  Randomization  Within  Test  Sessions 

Principles  of  good  experimental  design  require  that  all  the  test  combinations  shown  to  a 
viewer  be  randomly  permuted  over  the  viewer’s  four  session  tapes.  This  permits  each 
session  to  exhibit  a  full  range  of  video  quality  and  mitigates  artifacts  due  to  presentation 
order  (learning  or  fatigue  effects,  adjacency  effects,  etc). 

Ideally,  we  might  want  each  viewer  to  be  presented  with  an  independent  randomization 
of  the  stimuli,  but  we  are  constrained  by  the  need  to  prepare  each  session  as  a  pre-edited 
1/2  hour  video  tape,  so  we  plan  to  use  the  same  1/2  hour  tapes  (and  hence  the  same 
randomization  sequence  for  each  1/2  hour  tape)  for  all  the  viewers  presented  with  the 
same  selection  of  HRC’s.  The  presentation  order  for  the  four  session  tapes  will  be 
randomized  as  per  section  2.9.1. 

The  order  of  presentation  of  the  stimuli  can  influence  the  opinion  of  the  evaluators  in 
subtle  ways.  Therefore,  although  a  randomized  order  of  presentation  is  necessary,  it  may 
not  be  sufficient.  Following  the  CCIR  guidelines,  we  arranged  for  consecutive  stimuli  to 
be  dissimilar  on  each  of  the  two  design  dimensions,  meaning  that  not  only  the  pictorial 
content  but  also  the  transmission  impairments  caused  by  the  characteristics  of  the  HRC’s 
varies.  This  was  achieved  by  making  some  minor  modifications  to  the  randomization 
process  which  take  into  account  the  grouping  of  HRC’s  into  9  types,  and  a  grouping  of 
the  25  scenes  into  5  categories  with  similar  pictorial  content,  shown  in  Table  2. 

Table  2  identifies  each  scene  by  its  short  process  name  and  the  lower  case  letter  used  in 
the  test  matrix  on  Table  1. 
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Table  2.  SCENE  CONTENT  CATEGORIES 


CONTENT 

CATEGORY 

DESCRIPTION 

SCENE  NAMES  &  LETTERS 

A 

One  person, 
mainly  head  and 
shoulders 

vtclnw(f),  susie(j), 
disguy(k),  disgal(l) 

B 

One  person  with 
graphics  and/or 
more  detail 

vtcmp(a),  vtc2zm(b), 
boblec(e),  smityl(m), 
smity2(n),  vowels(w) 

C 

More  than  one 
person 

3inrow(d),  5rowl(g), 
intros(o),  3twos(p), 

2wbord(q),  split6(r) 

D 

Graphics  with 
pointing 

washdc(c),  cirkit(s), 
roadmap(i),  filter(u), 
ysmite(v),  inspec(x) 

E 

High  object  and/or 
camera  motion 
(Examples  of 
Broadcast  TV) 

fiogar(h),  ftball(i), 
fredas(y) 

Table  3  divides  the  25  HRC’s  into  9  groups  according  to  the  transmission  quality  and 
type  of  impairments  that  are  to  be  expected.  Some  revisions  may  be  desirable  after  the 
processed  tapes  have  been  reviewed. 

_ _  Table  3.  REFERENCE  CIRCUIT  GROUPS 


GROUP  NO. 

CIRCUIT  DESCRIPTION 

HRC  NO. 

1 

High  Quality 

1-3 

2 

Vector  Quantization,  medium  rate 

4-5 

3 

Proprietary,  low  to  medium  rate 

6-7 

4 

Proprietary,  medium  to  high  rate 

8-10 

5 

QCIF,  low  rate 

11-13 

6 

QCIF,  medium  rate 

14 

7 

CIF,  low  rate 

15-18 

8 

CBF,  medium  rate 

19-21 

9 

CIF,  high  rate 

22-25 

Each  processed  test  scene,  also  called  a  test  combination,  was  then  assigned  a  number 
and  letter  code  (such  as  5-B)  roughly  categorizing  pictorial  content  and  transmission 
circuit  characteristics. 

In  terms  of  these  categories  and  this  notation,  the  randomization  process  was  performed 
in  the  following  steps,  using  sampling  without  replacement,  subject  to  certain  constraints: 
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1.  All  test  scenes  processed  through  the  10  HRC’s  assigned  to  one  set  of  viewing 
tapes  were  put  into  a  pool  (the  number  10  includes  the  7  exclusively  assigned  to 
that  set,  plus  the  3  that  are  shared  across  sets). 

2.  Randomly,  a  test  scene  was  pulled  and  its  code  checked. 

3.  If  both  number  and  letter  were  different  from  the  preceding  scene,  it  was  accepted. 

4.  If  either  the  number  or  letter  were  the  same,  it  was  returned  to  the  pool  and  another 
scene  was  pulled,  until  one  was  found  that  was  accepted. 

5.  This  was  continued  until  all  spaces  on  the  tape  (either  64  or  65)  were  filled. 

6.  The  whole  process  above  was  then  continued  for  the  next  session  tape  in  the  set, 
using  the  remaining  combinations. 

This  selection  process  results  in  four  well-randomized  tapes  for  each  HRC  set.  We 
anticipated  some  difficulties  in  satisfying  the  adjacency  constraints  toward  the  end  when 
only  a  few  scenes  remained  in  the  pool.  Judicious  exchange  with  previously  assigned 
scenes  made  it  easy  to  solve  this  problem. 

With  64  or  65  stimuli  per  tape  and  only  10  HRC’s  per  tape,  the  above  described 
randomizing  process  produced  a  healthy  balance  of  HRC’s  across  the  four  session  tapes 
in  each  set. 

Special  attention  was  given  to  the  scenes  which  are  to  be  used  for  quality  checks  since 
their  number  is  held  to  a  minimum  to  avoid  excessive  disqualifications.  The  Null  circuit 
scenes  will  be  judged  by  viewers  primarily  in  terms  of  resolution  and  color  fidelity,  since 
motion  rendition  is  a  minor  factor.  Therefore,  scenes  washdc(c),  flogar(h),  cirkit(s),  and 
rodmap(t)  were  prime  candidates  for  this  purpose.  One  was  assigned  to  each  session 
tape.  Scenes  selected  for  repetition  were  typical  and  average  ones,  especially  those  with 
the  content/quality  code  8-C  or  9-D. 

The  location  of  Null  circuit  and  repeated  scenes  on  the  tape  were  not  determined  by  the 
randomization  process  described  above.  Instead,  locations  were  picked  judiciously,  by 
hand,  to  ensure  that  each  session  tape  contains  one  of  each  and  that  all  other  constraints 
were  satisfied.  These  locations  were  different  on  each  session  tape. 

The  above  randomization  prescriptions  produced  a  satisfactory  sequence  of  scenes  on 
each  tape,  but  one  further  level  of  randomization  was  adopted  to  further  neutralize 
potential  order-of-presentation  artifacts  —  a  block  randomization  achieved  by  presenting 
the  four  session  tapes  in  a  different  randomized  order  to  different  sets  of  1,  2,  or  3 
viewers,  as  described  in  Section  2.9.1. 

2.9  Procedures  for  Randomized  Tape  Viewing  and  Selection  of  Viewer  Groups 

The  following  guidelines  for  the  subjective  testing  laboratories  specify  how  to  divide  the 
pool  of  viewers  into  session  sub-teams  and  what  tape  presentation  order  to  use  for  each 
of  the  sub-teams.  The  intent  of  these  guidelines  is  to  minimize  systematic  differences 
which  could  lead  to  biases  in  HRC  ratings.  Whenever  possible,  the  testing  laboratories 
should  use  these  guidelines.  Any  exceptions  to  these  guidelines  will  be  recorded 
(Appendix  G  gives  an  alternative  method  for  random  selection  of  viewer  groups  that  will 
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be  used  at  one  or  more  labs).  Here  the  labs  are  identified  as  X,  Y,  Z  and  the  tapes  as  Rl, 
R2,  R3.  R4:  Gl.  G2.  G3.  G4;  OI,  02.  03,  04. 

2.9.1  Random  Ordering  of  Session  Tapes 

To  allow  for  the  possibility  of  tape  sharing  between  the  labs,  it  is  assumed  that  each  set 
of  four  tapes  will  be  viewed  in  succession  without  interruption  by  other  tapes.  The  order 
which  teams  view  the  tapes  is  determined  randomly  subject  to  balance.  Using  Tables  of 
Random  Permutations  by  L.E.  Moses  and  R.V.  Oakford  (Stanford  University  Press, 
1963)  yields  the  following  chronological  order  of  the  tape  sets: 

Table  4.  CHRONOLOGICAL  ORDER  OF  TAPE  SETS 
LabX:  GOR 
LabY:  ORG 
Lab  Z:  R  G  O 

Costs  permitting,  a  complete  set  of  tapes  will  be  prepared  for  each  lab  to  facilitate 
scheduling.  Each  of  the  9  teams  of  viewers  given  above  must  have  at  least  10  viewers. 
The  four  tapes  shown  to  each  of  the  nine  teams  are  to  be  ordered  in  a  random  fashion. 
Since  each  session  can  have  at  most  3  viewers,  at  least  4  random  tape  orderings  for  each 
group  of  10  viewers  is  possible.  For  all  9  groups,  this  gives  a  total  of  4  x  9  =  36  orders. 
There  are  41=24  permutations  of  the  integers  1, 2,3,4.  It  seems  reasonable  to  use  all  24 
permutations  and  include  random  duplicates  of  12  of  them.  This  is  done  by  using  Table 
1  of  Moses  and  Oakford,  which  consists  of  960  permutations  of  1,2,...,9,  ignoring  the 
numbers  5  through  9,  copying  the  successive  orders  of  1,2, 3 ,4,  deleting  any  order  that 
arises  more  than  twice,  and  accepting  exactly  12  duplicates  as  they  arise  until  36  are 
obtained: 


1423 

1234 

3142 

[4231] 


Table  5.  TAPE  ORDER  PERMUTATIONS 


2134 

3214 

1432 

3421 

2143 

3412 

3124 

4123 

[2314] 

[1342] 

4213 

[3124] 

[4321] 

1243 

2431 

2341 

4321 

[1423] 

4231 

1342 

[1234] 

2413 

2314 

[2134] 

[3412] 

[3142] 

[2413] 

[2431] 

1324 

4132 

4312 

3241 

Hence  the  orders  in  which  the  tapes  are  to  be  viewed  are  as  follows,  proceeding  from  left 
to  right  and  line  by  line: 


Table  6.  VIEWING  ORDER  FOR  SUB-TEAMS  BY  LAB 

LabX: 


Gl 

G4 

G2 

G3 

03 

04 

02 

01 

Rl 

R3 

R4 

R2 

LabY: 

03 

01 

02 

04 

R2 

R3 

Rl 

R4 

Gl 

G3 

G4 

G2 

G2 

Gl 

G3 

G4 

04 

03 

02 

01 

Rl 

R2 

R3 

R4 

04 

01 

02 

03 

R2 

Rl 

R3 

R4 

G4 

G2 

Gl 

G3 

G3 

G2 

Gl 

G4 

01 

04 

02 

03 

R2 

Rl 

R4 

R3 

01 

02 

03 

04 

R3 

Rl 

R4 

R2 

G3 

Gl 

G2 

G4 

Gl 

G4 

G3 

G2 

04 

02 

03 

01 

R3 

R4 

Rl 

R2 

02 

04 

Ol 

03 

R2 

R3 

Rl 

R4 

G3 

G4 

Gl 

G2 
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LabZ: 


R3 

R1 

R4 

R2 

R2 

R4 

R1 

R3 

R2 

R4 

R3 

R1 

R4 

R2 

R3 

R1 

G4 

G3 

G2 

G1 

G1 

G2 

G4 

G3 

G2 

G4 

G3 

G1 

G2 

G3 

G4 

G1 

01 

03 

02 

04 

04 

01 

03 

02 

04 

03 

01 

02 

03 

02 

04 

01 

2.9.2  Random  Selection  of  Viewer  Groups 

Each  lab  will  assemble  a  pool  of  viewers,  about  one-third  of  whom  will  view  the  R  tapes, 
another  third  the  G  tapes,  and  the  remaining  third  the  O  tapes.  Since  not  more  than  three 
viewers  will  participate  at  any  session,  there  will  be  at  least  four  sub-teams  of  viewers  of 
R  tapes,  four  of  G  tapes,  and  four  of  O  tapes  at  each  lab,  12  non-overlapping  sub-teams  at 
each  lab.  The  following  discussion  assumes  sub-teams  of  size  three.  Even  if  all 
members  of  a  sub-team  do  not  view  the  tapes  at  the  same  time,  they  must  view  them  in 
the  same  order. 

The  viewers  shall  be  assigned  to  sub-teams  at  random  to  avoid  systematic  differences 
between  the  R,  G,  and  O  teams,  which  could  lead  to  biases  in  HRC  ratings.  Each  lab 
shall  list  its  viewers  in  alphabetical  order  and  assign  the  numbers  1,2,...  in  that  order. 

Three  tables  of  random  permutations  of  1,2,. ..,50  from  Moses  and  Oakford  are  shown 
below.  Lab  X  is  to  use  the  first  permutation,  Y  the  second,  and  Z  the  third,  going  down 
the  columns  and  disregarding  any  numbers  beyond  the  available  number  of  viewers.  If 
Lab  X  has  36  viewers,  then  its  first  sub-team  has  viewers  No.  8,  3,  5;  its  second  No.  13, 
10. 27;  its  third  No.  34,  12. 25;  its  fourth  No.  14. 4. 23;  its  fifth  No.  15.  18,  2;  etc. 

This  means  that  the  first  sub-team  at  Lab  X,  viewers  8,  3,  5,  will  view  the  G  tapes  in  the 
order  G1  G4  G2  G3;  the  second  sub-team  of  Lab  X,  viewers  13,  10,  27,  will  view  the  G 
tapes  in  the  order  G2  G1  G3  G4;  the  third  sub-team  in  the  order  G3  G2  G1  G4;  and  the 
fourth  sub-team  in  the  order  G1  G4  G3  G2.  The  fifth  sub-team  at  Lab  X  will  view  the  O 
tapes  in  the  order  03  04  02  Ol,  etc.,  up  to  the  twelfth  sub-team  at  Lab  X,  which  will 
view  the  R  tapes  in  the  order  R3  R4  R1  R2. 

Similarly  Lab  Y  will  choose  12  sub-teams  according  to  the  second  permutation.  If  it  has 
37  viewers  available,  its  first  sub-team  of  three  consists  of  viewers  No.  24,  16,  36;  its 
second.  No.  21,  15, 2;  its  third.  No.  1 1, 20,  31;  etc. 

Similarly  Lab  Z  will  choose  12  sub-teams  according  to  the  third  permutation.  If  it  has  36 
viewers  available,  its  first  sub-team  consists  of  viewers  No.  22, 9,  15;  etc. 

Table  7.  RANDOM  VIEWER  ASSIGNMENT  TO  SUB-TEAMS 


8 

27 

4 

2 

26 

48 

38 

46 

9 

39 

3 

34 

23 

21 

6 

50 

37 

40 

16 

19 

5 

12 

15 

7 

32 

49 

30 

42 

11 

31 

13 

25 

18 

43 

28 

22 

29 

41 

1 

35 

10 

14 

44 

20 

36 

47 

17 

45 

33 

24 

LabX 
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Lab  Y 

24  2  31  19  46  17  37  13  28  33 

16  41  3  43  1  7  18  30  39  27 


36 

11 

10 

8 

44 

26 

12 

23 

47 

34 

21 

49 

50 

38 

25 

4 

5 

32 

9 

40 

15 

20 

6 

45 

22 

29 

14 

42 

35 

48 

22 

15 

31 

12 

5 

24 

14 

29 

45 

30 

39 

44 

43 

41 

16 

8 

21 

35 

49 

37 

40 

18 

28 

47 

11 

33 

36 

20 

42 

26 

9 

38 

7 

17 

46 

25 

23 

2 

4 

19 

48 

13 

50 

34 

6 

10 

32 

27 

3 

1 

3.  PRE-TEST  PROCEDURES 

1.  All  viewers  will  complete  a  pre-test  questionnaire  (See  Appendix  C). 

Note:  The  subject  demographic  data  will  be  examined  only  if  the  need  arises,  such 
as  if  differences  emerge  in  the  Lab  to  Lab  calibration  checks  and  possible  reasons 
are  sought  for  the  differences.  However,  the  3  test  labs  will  need  to  examine  the 
question  on  video  telephony  usage,  to  ensure  that  they  have  met  the  sample 
experience  requirement. 

2.  All  viewers  will  complete  vision  acuity  tests  at  the  (6H)  viewing  distance,  and 
color  tests  at  the  distance  recommended  in  the  color  test  procedure.  The  specific 
test  for  acuity  is  a  Graham-Field  Catalog  #  13-1240  single  letter  identification 
chart,  or  equivalent,  and  modified  for  use  at  6H  distance.  The  specific  test  for  color 
vision  is  the  Pseudo-Isochromatic  Plates  for  Testing  Color  Perception,  as  supplied 
by  Beck  Engraving  Co.,  for  example. 

3.  At  each  session.  Labs  will  record  the  session  number,  tape  number,  viewing 
position(s),  and  time  of  day. 

4.  Instructions  must  be  standardized  and  delivered  on  tape.  Appendix  A  gives  the 
text  of  the  Instructions. 

5.  There  will  be  a  practice  session,  consisting  of  6  cycles  through  the  viewing  and 
voting  process.  The  range  of  quality  displayed  to  the  subjects  will  represent  nearly 
the  full  range  of  quality  in  the  experiment  and  will  also  contain  one  combination 
with  transmission  errors.  In  the  spirit  of  conforming  with  CCIR  500-5,  there  will 
be  a  2  minute  break  between  practice  and  test  sessions. 

6.  During  subsequent  sessions,  the  subjects  will  again  view  the  practice  tape  sequence 
prior  to  the  new  material.  There  should  be  no  need  to  practice  voting  in  these 
sessions. 


4.  TEST  SESSIONS 
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4.1  Considerations 

1.  Test  scene  numbers  must  be  announced  prior  to  the  sequence. 

2.  Audio  cues  to  vote  will  be  included. 

3.  Use  voting  forms  as  shown  in  Appendix  B. 

4.2  Session  Scheduling 

The  scheduling  of  the  test  sessions  (time  of  day,  rest  between  sessions,  etc.)  can  affect  the 
test  results.  Compliance  with  CCIR  500-5  and  availability  of  test  personnel  can  produce 
significant  constraints  on  the  time  at  which  each  test  is  performed. 

Scheduling  of  individual  test  sessions,  in  terms  of  the  number  of  sessions  per  week  per 
subject  is  left  to  each  individual  lab’s  discretion.  There  will  be  a  15  minute  break 
between  sessions  in  a  pair,  and  a  1  minute  break  during  each  session.  There  will  be  at 
least  1/2  day  rest  between  pairs  of  test  sessions. 

Test  personnel  should  not  change  between  sessions. 

4.3  Viewing  Conditions 

In  preparing  the  viewing  conditions,  the  test  labs  will  use  Section  2  of  CCIR  500-5  as  a 
guideline.  All  viewing  will  be  conducted  at  a  distance  equal  to  6  picture  heights  (6H). 

The  viewing  monitor  used  at  each  test  lab  will  be  the  SONY  BVM-1910  or  equivalent. 

The  format  of  the  session  tapes  will  be  Betacam  SP. 

The  SMPTE  color  bars  alignment  signal  will  be  available  on  tape. 

5.  LNITIAL  ANALYSIS  OF  SUBJECTIVE  DATA 

The  purpose  of  this  section  is  to  define  the  data  processing  and  initial  analysis  that  will  be 
performed  on  the  data  obtained  from  the  subjective  tests.  The  steps  described  in 
subsections  5.1  tlirough  5.6  are  to  be  performed  at  each  lab  on  the  data  obtained  at  that 
lab. 

5.1  Data  Entry 

The  inputs  to  the  Data  Analysis  function  are  the  score  sheets  filled  in  by  the  test 
observers  (viewers)  at  each  of  the  3  labs.  At  each  lab,  there  will  be  30  primary  viewers, 
allocated  into  3  teams  of  10  each,  and  possibly  some  number  of  alternates  or 
replacements  for  each  team.  The  teams  are  coded  as  Red,  Green  and  Orange, 
corresponding  to  the  color-coding  of  the  video  tapes  they  will  view.  The  tapes  come  in 
sets  of  4  for  the  4  sessions  that  each  viewer  will  attend.  For  each  viewer  and  each  session 
there  will  be  64  or  65  opinion  scores.  Each  score  will  be  assigned  an  integer  from  1  to  5. 
For  the  total  experiment  there  will  be  23,220  opinion  scores  (not  counting  disqualified 
viewer  and  surplus  alternates).  The  original  score  sheets  will  be  duplicated,  and  the 
original  sheets  will  be  stored  at  a  central  location  (place  TBD).  All  further  processing 
will  be  performed  on  the  copies. 
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The  data  from  each  score  sheet  will  be  keyed  into  a  computer.  The  data  on  each  sheet 
consists  of  the  test  observer  ID,  the  lab  ID,  the  session  number,  and  64  or  65  opinion 
scores.  Data  will  be  keyed  in  by  two  independent  operators,  and  a  computer  program 
will  verify  that  the  entered  data  is  consistent. 

Laboratories  should  coordinate  the  keying  procedure  to  simplify  interlab  exchange  of 
data  (e.g.,  ASCII). 

5-2  Convert  Presentation  Sequence 

The  randomization  sequences  for  each  lab,  as  shown  in  Appendix  F,  will  be  made 
available  by  Delta  Information  Systems  in  WordPerfect  format.  These  sequences  will  be 
converted  by  each  lab  to  a  format  that  can  be  used  for  its  computer  program. 

5.3  Perform  Null  and  Repeat  Quality  Checks 

Each  lab  will  analyze  its  own  data  to  determine  the  quality  of  the  scores  given  by  each 
test  observer. 

In  each  test  session  there  is  a  Null  (designated  #0  in  Appendix  F)  inserted  to  test  the 
observer.  If  on  any  of  these  Nulls  the  observer  gives  a  score  of  3.0  or  less,  the  observer 
will  be  disqualified. 

In  each  test  session  one  combination  of  HRC  and  scene  is  repeated.  If  for  any  repeated 
combination  the  absolute  difference  between  the  two  opinion  scores  given  by  a  test 
observer  is  3.0  or  more,  the  observer  will  be  disqualified. 

Alternate  viewers  for  each  team  (if  any)  are  initially  marked  as  disqualified.  Some 
number  of  them  will  subsequently  be  marked  as  qualified  if  they  pass  the  two  quality 
checks  and  if  they  are  needed  to  replace  disqualified  primary  team  members. 

5.4  Reformat  the  Data 

Using  the  keyed-in  score  data  and  the  presentation  sequences  for  each  session,  each  lab 
will  use  a  computer  spreadsheet  program  to  reformat  the  data  into  a  series  of  tables  as 
shown  below. 

In  each  session,  there  is  one  combination  that  is  repeated  twice;  the  score  from  the  first 
occurrence  shall  be  entered  into  the  main  body  of  the  table,  and  the  second  shall  be 
entered  into  the  "REPEAT"  column. 

5.5  Compute  Summary  Statistics  Per  HRC 

For  each  of  the  reformatted  tables  (HRCs)  obtained  from  Section  5.4,  the  following 
calculations  are  performed  by  computer  at  each  lab: 

The  average  and  sample  standard  deviation  of  each  column  (scene)  are  calculated  across 
all  qualifying  rows,  and  displayed  to  two  decimal  places  in  the  table. 

The  standard  error  of  the  mean  of  each  column  is  calculated  as  the  standard  deviation 
calculated  above  divided  by  the  square  root  of  the  number  of  qualifying  values  in  the 
column,  and  displayed. 


-  15- 


T1AI.5/94-118R1 


LAB  _  HRC  NO.  _ 


--  VIEWER  --  - - SCENE . . . REPEAT  -- 

TEAM  ID  QUALIFY?  a  b  c  d  e...w  x  y  SCENE  SCORE 


RED 

1 

1 

2 

1 

10 

1 

(ALT.) 

11 

0 

12 

0 

20 

0 

GREEN 

21 

1 

22 

1 

30 

1 

(ALT.) 

31 

0 

32 

0 

40 

0 

ORANGE 

41 

1 

42 

1 

50 

1 

(ALT. ) 

51 

0 

52 

0 

60 

0 

MEAN 

S.D. 

S.E. 

MAX 

MIN 

The  maximum  and  minimum  qualifying  value  in  each  column  are  determined  and 
displayed. 

5.6  Lab-to-Lab  Analysis 

Up  to  this  point,  all  analysis  will  be  performed  by  each  lab  on  the  data  that  it  has 
gathered.  From  this  point  on,  the  data  will  be  analyzed  as  a  whole.  The  results  obtained 
from  Sections  5.4  and  5.5  will  be  distributed  to  ail  concerned  parties  for  further  analysis. 

Qualifying  data  from  the  various  labs  will  be  studied  for  interlab  consistency,  and  pooled 
in  a  statistically  suitable  fashion,  to  produce  the  tabulation  below  for  use  in  the  next 
(objective)  phase  of  the  study. 

Plans  for  further  data  analysis  will  be  presented  in  future  T1A1.5  contributions. 
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TIA1.5/94-118RI 


SUBJECTIVE  TEST  RESULTS 


SCENE  a  b  c  d  e  .  x  y 

HRC 

1  mean 

s  .d. 

s  •  e . 

max 

min 

2  .  mean 

s .  d. 

S.e, 

max 

min 

3 

25  mean 

s  .d. 

s.e. 

max 

min 

NOTE:s.d.  estimate  of  standard  deviation  of  observer  population 

s.e.  estimate  of  standard  error  of  mean 

max  largest  observed  opinion  score 

min  smallest  observed  opinion  score 


6.  AD  HOC  GROUP  MEMBERSHIP 

The  complete  list  of  Detailed  Test  Plan  Ad  Hoc  Group  members  and  Data  Analysis  Plan 
Ad  Hoc  Group  members  can  be  found  in  Appendix  E. 
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APPENDEXA  •  Instructions  to  Test  Subjects 


Instructions  for  Initial  Session 

The  ANSI  Standards  Committee  on  Telecommunications  is  conducting  a  study  to 
determine  how  different  video  telephony  systems  affect  the  delivered  quality.  Today  we 
are  asking  you  to  help  us  measure  the  quality  of  the  video  scenes  you  are  about  to  see. 

We  are  not  asking  you  to  rate  the  content  of  the  scenes,  the  artistic  composition  nor  the 
quality  of  the  acting.  Rather,  we  are  asking  for  your  rating  of  the  quality  of  the  video 
image  itself. 

You  will  be  shown  two  versions  of  the  same  video  scene.  The  first  version  will  be  the 
original  scene.  The  second  version  will  be  the  original  passed  through  a  video 
transmission  system.  You  are  asked  to  rate  the  difference  in  quality  that  you  perceive 
between  the  two  versions.  As  shown  on  your  rating  form,  please  score  the  differences  as 
either  Imperceptible  (when  you  cannot  see  a  difference  between  the  first  and  second 
versions).  Perceptible  but  not  Annoying,  Slightly  Annoying,  Annoying,  or  Very 
Annoying. 

You  should  mark  your  rating  form  as  indicated  on  the  practice  sheet.  Note  that  response 
ovals  for  each  scene  pair  are  arranged  in  a  vertical  column  below  the  scene  number. 
Mark  only  one  oval  for  each  scene  pair. 

Remember,  there  are  no  right  and  wrong  answers.  We  are  interested  in  how  you, 
personally,  perceive  the  difference  between  the  two  versions.  It  is  not  necessary  to  think 
long  about  your  answer.  However,  please  watch  the  entire  scene  before  scoring.  Please 
do  not  discuss  the  scenes  with  your  fellow  viewers.  Your  first  reaction  is  what  we  wish 
you  to  record. 

We  now  begin  the  Practice  Section.  Use  the  your  rating  form  labeled  "Practice  Section" 
and  mark  the  appropriate  oval  for  the  following  six  pairs  of  scenes.  The  practice  session 
displays  representative  examples  of  video  quality  in  the  test. 

The  announcer  will  give  the  number  of  each  pair  of  scenes  before  it  appears.  After 
viewing  each  scene  pair,  the  announcer  will  ask  you  to  rate  the  video  quality. 

We  now  begin  the  rating  process  for  the  Practice  Section. 
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Here  is  scene  1.  (speak  quickly) 


Please  score  scene  1. 


Here  is  scene  6. 


Please  score  scene  6. 

That  completes  the  Practice  Section.  We  will  begin  the  First  Section  in  a  few  moments. 
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Instructions  for  Follow-on  Sessions 
Welcome  back  to  the  Video  Viewing  Lab. 

Today’s  session  will  be  similar  to  your  last  session  with  us. 

You  will  be  shown  two  versions  of  the  same  video  scene.  The  first  version  will  be  the 
original  scene.  The  second  version  will  be  the  original  passed  through  some  video 
system.  You  are  asked  to  rate  the  difference  in  quality  that  you  perceive  between  the  two 
versions.  As  shown  on  your  rating  form,  please  score  the  differences  as  either 
Imperceptible  (when  you  cannot  see  a  difference  between  the  first  and  second  versions), 
Perceptible  but  not  Annoying,  Slightly  Annoying,  Annoying,  or  Very  Annoying. 

Please  watch  the  entire  scene  before  scoring,  and  do  not  discuss  the  scenes  with  your 
fellow  viewers.  Your  first  reaction  is  what  we  wish  you  to  record. 

The  announcer  will  give  the  number  of  each  pair  of  scenes  before  it  appears.  After 
viewing  each  scene  pair,  the  announcer  will  ask  you  to  rate  the  video  quality. 

We  now  begin  the  Practice  for  the  First  Section.  You  should  mentally  score  the  scenes  in 
this  sequence,  but  do  not  mark  your  rating  form  at  this  time. 
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Instnictions  for  Viewing  Session  Tapes 
We  now  begin  the  rating  process  for  the  First  Section. 
Here  is  scene  1. 


Please  score  scene  1. 


Here  is  scene  30. 


Please  score  scene  30. 

That  completes  the  First  Section.  We  will  begin  the  Second  Section  in  a  few  moments. 
(60  second  break) 

We  now  begin  the  rating  process  for  the  Second  Section. 

( "Here  is  scene  31"  and  so  on  to  scene  64  or  65  ) 

That  completes  the  Second  Section. 

This  concludes  today’s  video  quality  rating  session.  We  look  forward  to  seeing  you  again 
for  the  next  video  quality  rating  session. 
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APPENDIX  B  •  PRACTICE  and  VOTING  FORMS 

Viewer  Rating  Practice  Form 


You  will  be  shown  two  versions  of  the  same  video  scene.  The  first  version  will  be  the  original  scene. 
The  second  version  will  be  the  original  passed  through  some  video  system.  Rate  the  difference  between 
the  two  versions  by  marking  one  of  the  five  ovals.  See  the  examples  below  which  show  typical 
markings  for  3  scene  pairs. 


Marking  Examples 


Imperceptible 

10 

11 

iP; 

Perceptible  but  not  Annoying 

O 

Slightly  Annoying 

O 

V-  .. 
|0- 

Annoying 

o 

m" 

Very  Annoying 

o 

<c>- 

Scene  Pair  Numbers 

12  ^ 

O  Imperceptible 
^  Perceptible  but  not  Annoying 
(3  Slightly  Annoying 
Q  Annoying 
O  ^ery  Annoying 


Practice  Scenes 

1 

2 

3 

4  5 

6 

Imperceptible 

O 

Pi 

o 

O 

Imperceptible 

Perceptible  but  not  Annoying 

:q 

O 

::P 

o  p 

o 

Perceptible  but  not  Annoying 

Slightly  Annoying 

O 

P' 

o  P' 

o 

Slightly  Annoying 

Annoying 

o 

O 

•o  ■ 

O  ifO' 

o 

Annoying 

Very  Annoying 

c5 

O 

o 

o  o 

o 

Very  Annoying 
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APPENDIX  B  •  PRACTICE  and  VOTING  FORMS 


1 


1 

2 

3 

4 

5 

6 

Imperceptible 

-o 

O 

:0 

O 

rO 

O 

Perceptible  but  not  Annoying 

O 

O- 

O 

So 

o 

Slightly  Annoying 

s' 

^•o 

O 

O 

jO' 

o 

Annoying 

io 

O 

o- 

X  :*>  •• 

O 

fo 

o 

Very  Annoying 

Id' 

O 

<•  • 
\d; 

O 

o 

Date -  Time _  Evaluator  No _ Viewing  Position 

Signature  _ 

(Name) 


Imperceptible 

Perceptible  but  not  Annoying 
Slightly  Annoying 
Annoying 
Very  Annoying 


Tape  No. 


-23- 


APPENDIXC  -  PRE-TEST  QUESTIONNAIRE 

Please  answer  the  following  questions:  NAME _ 

1.  Circle  your  age  range: 

10-19  20-29  30-39  40-49  50-59  60-69  70-79  80-89 


2. 


j. 


Circle  your  Gender: 


Male  Female 

Indicate  your  occupation  according  to  industry  and  job:  (Mark  one  in  each  column) 


INDUSTRY _ 

_Accounting/Legal/Consulting 
_Advertising/Public  Relations 
_AgricuI  ture/Forestry 
_Broadcasting/Newspapers 
_Construction/Contracting 
_Education 

_Electronics/Computers 
_Engineering/Architecture 
_Entertainment 
_F  i  nance/I  nsurance 
Government 

_Health  Care/Social  Services 
Manufacturing/Printing 
_Military 
_Mining/Oil/Gas 
Personal/Business  Services 
Real  Estate 
_Religion 

Restaurants/Lodging 

_RetaU 

_Retired 

Security/Services/Police 

TelecommunicationAJtilities 

Transportation 

.Wholesale 

None 


_ JOB _ 

AdministrativeAVhite  Collar 
_Clerical/Support 
_Driver 

Executive/Managerial 
_Farmer/Forester 
Homemaker 
Industrial  Worker 
Laborer 
Maintenance  . 
Owner/Operator 
Police/Military 
.Professional 
Rancher/Fisher 
Sales/Marketing 
Secretarial 

Semi-Professional  Services 

Skilled  Trades 

Student 

Technical 

Unemployed 


4.  Have  you  previously  participated  in  video  telephony,  or  in  a  video  teleconference? 
(Mark  one  answer) 

—  Yes,  within  the  last  2  years. 

—  Yes,  but  not  within  the  last  2  years. 

—  No. 
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APPENDIX  D  -  TABLE  OF  HYPOTHETICAL  REFERENCE  CIRCUITS 

This  table  first  appeared  in  document  TlAl. 5/92-174.  As  permited  in  their  charter,  the 
Testing  Ad  Hoc  Group  (H.  Meiseles,  Vyvx,  Chair;  S.  Gallaher,  Vyvx;  A.  Morton,  AT&T 
Communications)  modified  the  table  slightly  to  comply  with  the  limitations  of  the 
available  equipment.  The  modified  version  appears  below. 

HYPOTHETICAL  REFERENCE  CIRCUITS 


HRC 

Algorithm 

(vendor) 

Resolution 

Total, 

Kbps 

Audio, 

Kbps 

Video, 

Kbps 

Coding 

Mode 

Frame 

Rate 

FEC 

Burst 

Errors 

n 

Null 

- 

- 

- 

- 

- 

. 

- 

Off 

VHS 

- 

- 

- 

- 

- 

- 

- 

Off 

mm 

Proprietary 

V.High 

45,000 

- 

- 

- 

- 

- 

Off 

B 

Proprietary 

Med. 

128 

- 

- 

VQ 

. 

. 

Off 

Bi 

Proprietary 

High 

336 

- 

- 

VQ 

- 

- 

Off 

6 

Proprietary 

Med. 

112 

- 

- 

- 

- 

- 

Off 

7 

Proprietary 

Med. 

384 

- 

- 

- 

- 

- 

Off 

8 

Proprietary 

Med. 

768 

- 

- 

- 

- 

- 

Off 

9 

Proprietary 

High 

768 

- 

- 

- 

- 

- 

Off 

Proprietary 

High 

1536 

- 

- 

- 

- 

- 

Off 

wm 

H.261(dif0 

QQF 

128 

56 

70.4 

INTER+MC 

- 

On 

Off 

B 

H.261(same) 

QQF 

128 

56 

70.4 

INTER 

O 

* 

On 

Off 

IB 

H.261(same) 

QQF 

168 

48 

118.4 

INTER+MC 

• 

On 

Off 

14 

H.261(di£f) 

QQF 

384 

56 

326.4 

INTER+MC 

- 

On 

Off 

15 

H.261(same) 

QF 

112 

48 

INTER+MC 

- 

On 

Off 

16 

H.261(same) 

QF 

128 

56 

70.4 

INTER+MC 

- 

■SB 

Off 

17 

H.261(dif0 

QF 

128 

48 

78.4 

INTER+MC 

- 

Off 

18 

H.261(same) 

QF 

168 

48 

118.4 

INTER+MC 

- 

Off 

19 

H.261(same) 

OF 

256 

56 

190.4 

INTER+MC 

15* 

On 

On 

20 

H.261(same) 

QF 

384 

56 

326.4 

INTER+MC 

- 

On 

Off 

21 

H.261(same) 

QF 

384 

56 

INTER+MC 

- 

On 

On 

22 

H.261(diff) 

QF 

768 

56 

BBB 

INTER+MC 

- 

On 

Off 

23 

H.261(same) 

QF 

768 

56 

710.4 

INTER+MC 

- 

On 

On 

24 

H.261(dif0 

QF 

1536 

56 

1478.4 

INTER+MC 

- 

On 

Off 

25 

H.261(same) 

QF 

1536 

56 

1478.4 

INTER+MC 

- 

On 

Off 

•  Specified  value.  Actual  frame  rate  may  be  determined  through  measurement. 


APPENDIX  E  .  DETAILED  TEST  PLAN  AD  HOC  GROUP  MEMBERS 


NAME  REPRESENTING 

AI  Morton  AT&T  Communications  (Chair) 

Tony  Schiano  AT&T  Communications 

David  Hayner  Ameritech  Srvcs 

Keith  Kommeyer  Bell  Atlantic 

Ron  McConnell  BellCore 

DanWirth  BellCore 

Dan  Klenke  Compression  Labs,  Inc 

R.  Schaphorst  Delta  Information  Sys 

Neil  Randall  Delta  Information  Sys 

John  Roth  Delta  Information  Sys 

Greg  Cermak  GTE  Labs 

Eric  Hauch  Government  of  Canada 

Stephen  Wolf  ^TIAy^TS.N3 

Rich  Baker  PictureTel  Corp 

Robert  Reynolds  (formally)  PictureTel  Corp 

Xian-Cheng  Yuan  PictureTel  Corp 

Greg  Onyszchuk  Telecom  Canada 

Doug  Stevens  Tektronix 

John  Grigg  US  West 

Joe  Duran  VTEL 

DATA  ANALYSIS  AD  HOC  GROUP  MEMBERS 

NAME  REPRESENTING 

Richard  Schaphorst  Delta  Information  Sys  (Convener) 

Bill  Coufal  US  West  (Secretary) 

Al  Morton  AT&T  Communications 

Paul  Tukey  BellCore 

DanWirth  BellCore 

CFrank  Taylor  Bell  South 

Neil  Randall  Della  Information  Sys 

Greg  Cermak  GTE  Labs 

Eric  Hauch  Government  of  Canada 

Edwin  Crow  NTIA/ITS.N3 

Arthur  Webster  NTIA/ITS.N3 

Stephen  Wolf  NTIA/ITS.N3 

Rich  Baker  PictureTel  Corp 

Marshall  Schachtman  PictureTel  Corp 
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APPENDIX  F  -  RANDOM  PRESENTATION  ORDERS 

PRESENTATION  ORDER  —  RED  TEAM  TAPES 

HRCs  1  4  7  8  13  15  19  20  22  24 


SESSION  1 

SESSION  2 

SESSION  3 

SESSION  4 

Scene/Type 

Scene/Type 

Scene/Type 

Scene/Type 

1 

13-f  5-A 

19-k  8-A 

22-n 

9-B 

15-s  7-D 

2 

20-i  8-E 

1-s  1-D 

4-f 

2-A 

13-q  5-C 

3 

4-q  2-C 

7-b  3-B 

7-u 

3-D 

7-a  3-B 

4 

19-x  8-D 

19-j  8-A 

8-0 

4-C 

8-q  4-C 

5 

15-1  7-A 

*22-c  9-D 

22-t 

9-D 

19-m  8-B 

6 

8-r  4-C 

19-b  8-B 

7-k 

3-A 

4-s  2-D 

7 

15-e  7-B 

1-k  1-A 

4-v 

2-D 

22-i  9-E 

8 

1-p  1-C 

24-p  9-C 

8-y 

4-E 

8-w  4-B 

9 

19-n  8-B 

4-y  2-E 

19-1 

8-A 

20-s  8-D 

10 

4-r  2-C 

19-t  8-D 

8-t 

4-D 

24-e  9-B 

11 

24-c  9-D 

7-y  3-E 

20-w 

8-B 

15-p  7-C 

12 

8-e  4-B 

4-w  2-B 

7-f 

3-A 

22-v  9-D 

13 

7-q  3-C 

#0-t  1-D 

20-n 

8-B 

15-j  7-A 

14 

15-m  7-B 

13-i  5-E 

7-h 

3-E 

20-y  8-E 

15 

13-g  5-C 

24-1  9^A 

4-d 

2-C 

#0-S  1-D 

16 

22-k  9-A 

1-b  1-B 

22-s 

9-D 

13-1  5-A 

17 

8-g  4-C 

22-u  9-D 

4-a 

2-B 

24-d  9-C 

18 

15-x  7-D 

7-1  3-A 

*19-p 

8-C 

8-in  4-B 

19 

1-r  1-C 

19-d  8-C 

24-n 

9-B 

24-r  9-C 

20 

7-t  3-D 

22-b  9-B 

19-0 

8-C 

13-a  5-B 

21 

13-w  5-B 

4-x  2-D 

1-t 

1-D 

19-h  8-E 

22 

15-f  7-A 

19-r  8-C 

8-a 

4-B 

13-d  5-C 

23 

1-w  1-B 

22-x  9-D 

20-f 

8-A 

19-u  8-D 

24 

8-v  4-D 

8-p  4-C 

24-g 

9-C 

7-i  3-E 

25 

4-h  2-E 

4-j  2-A 

8-i 

4-E 

19-v  8-D 

26 

22-1  9-A 

7-r  3-C 

13-c 

5-D 

24-h  9-E 

27 

20-x  8-D 

*22-0  9-D 

7-0 

3-C 

7-d  3-C 

28 

22-g  9-C 

7-p  3-C 

1-v 

1-D 

*24-x  9-D 

29 

8-j  4-A 

8-C  4-D 

8-1 

4-A 

13-y  5-E 

30 

22-h  9-E 

7-w  3-B 

1-m 

1-B 

7-n  3-B 

31 

*20-0  8-C 

24-y  9-E 

15-u 

7-D 

24-u  9-D 

32 

13-e  5-B 

20-p  8-C 

22-e 

9-B 

7-j  3-A 

33 

15-i  7-E 

4-u  2-D 

13-j 

5-A 

15-h  7-E 

34 

20-1  8-A 

24-a  9-B 

20-r 

8-C 

20-c  8-D 

35 

15-b  7-B 

4-i  2-E 

1-h 

1-E 

24-0  9-C 

36 

19-y  8-E 

8-k  4-A 

8-s 

4-D 

7-e  3-B 

37 

15-t  7-D 

15-g  7-C 

1-g 

1-C 

24-t  9-D 

38 

1-e  1-B 

13-x  5-D 

8-u 

4-D 

20-m  8-B 

39 

15-0  7-C 

4-n  2-B 

15-a 

7-B 

24-j  9-A 

40 

24-v  9-D 

1-q  1-C 

8-f 

4-A 

7-in  3-B 

41 

1-d  1-C 

22-w  9-B 

20-g 

8-C 

24-k  9-A 

42 

24-b  9-B 

19-c  8-D 

1-a 

1-B 

20-a  8-B 

43 

8-d  4-C 

24-f  9-A 

7-s 

3-D 

*24-x  9-D 

44 

20-u  8-D 

1-y  1-E 

4-b 

2-B 

13-0  5-C 

45 

24-i  9-E 

13-t  5-D 

*19-p 

8-C 

4-1  2-A 
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46 

O 

I 

O 

1-D 

24-q 

9-C 

i-f 

1-A 

15-y 

7-E 

47 

19-.q 

8-C 

20-b 

8-B 

19>s 

8-D 

19-f 

8 -A 

48 

7-v 

3-D 

1-x 

1-D 

24-w 

9-B 

4-m 

2-B 

49 

19-i 

8-E 

8-n 

4-B 

15-v 

7-D 

22-q 

9-C 

50 

*20-0 

8-C 

15-d 

7-C 

22-p 

9-C 

1-j 

1-A 

51 

1-n 

1-B 

22-a 

9-B 

13-u 

5-D 

20-q 

8-C 

52 

4-t 

2-D 

7-c 

3-D 

#0-h 

1-E 

22-in 

9-B 

53 

13-h 

5-E 

4.p 

2-C 

8-b 

4-B 

4-k 

2 -A 

54 

20-t 

8-D 

1-u 

1-D 

4-g 

2-C 

15-w 

7-B 

55 

24-m 

9-B 

4-e 

2-B 

22-j 

9 -A 

1-0 

1-C 

56 

20-v 

8-D 

24-s 

9-D 

8-h 

4-E 

15-c 

7-D 

57 

13-r 

5-C 

13-b 

5-B 

15-r 

7-C 

1-i 

1-E 

58 

19-e 

8-B 

19-g 

8-C 

1-1 

1-A 

7-x 

3-D 

59 

15-k 

7-A 

1-c 

1-D 

13-in 

5-B 

22-y 

9-E 

60 

4-c 

2-D 

13-k 

5 -A 

4-0 

2-C 

20-j 

8 -A 

61 

20-e 

8-B 

22-0 

9-C 

20-k 

8 -A 

13-s 

5-D 

62 

7-g 

3-C 

15-n 

7-B 

13-v 

5-D 

20-h 

8-E 

63 

8-x 

4-D 

13-p 

5-C 

19-w 

8-B 

22-r 

9-C 

64 

20-d 

8-C 

22-f 

9 -A 

15-g 

7-C 

13-n 

5-B 

65 

19-a 

8-B 

22-d 

9-C 
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PRESENTATION  ORDER  —  GREEN  TEAM  TAPES 


HRCs  2  5  6  10  14  15  16  17 


SESSION  1 

SESSION  2 

Scene/Type 

Scene/Type 

1 

2-v  1-D 

2-e  1-B 

2 

,  17-k  7-A 

17-g  7-C 

3 

14-u  6-D 

5-x  2-D 

4 

15-e  7-B 

16-g  7-C 

5 

2-h  1-E 

#0-c  1-D 

6 

23-e  9-B 

15-h  7-E 

7 

6-q  3-C 

14-x  6-D 

8 

17-y  7-E 

16-r  7-C 

9 

2-p  1-C 

23-x  9-D 

10 

15-v  7-D 

16-k  7-A 

11 

14-y  6-E 

10-i  4-E 

12 

10-w  4-B 

14-g  6-C 

13 

23-g  9-C 

20-m  8-B 

14 

5-f  2-A 

15-x  7-D 

15 

14-h  6-E 

14-n  6-B 

16 

*20-d  8-C 

17-u  7-D 

17 

6-c  3-D 

14-j  6-A 

18 

16-n  7-B 

5-s  2-D 

19 

5-g  2-C 

17-n  7-B 

20 

6-e  3-B 

23-i  9-A 

21 

5-0  2-C 

20-g  8-C 

22 

17-e  7-B 

16-a  7-B 

23 

20-f  8-A 

20-i  8-E 

24 

17-0  7-C 

16-j  7-A 

25 

6-v  3-D 

2-b  1-B 

26 

20-a  8-B 

10-d  4-C 

27 

6-p  3-C 

6-h  3-E 

28 

20-s  8-D 

14-d  6-C 

29 

23-k  9-A 

20-v  8-D 

30 

5-v  2-D 

10-a  4-B 

31 

2-i  1-E 

15-c  7-D 

32 

5-p  2-C 

5-i  2-E 

33 

23-y  9-E 

2-k  1-A 

34 

20-e  8-B 

17-h  7-E 

35 

23-1  9-A 

23-v  9-D 

36 

15-a  7-B 

14-i  6-E 

37 

10-v  4-D 

17-f  7-A 

38 

15-p  7-C 

10-c  4-D 

39 

20-k  8-A 

16-m  7-B 

40 

6-t  3-D 

*23-u  9-D 

41 

15-k  7-A 

10-e  4-B 

42 

14-r  6-C 

14-s  6-D 

43 

10-b  4-B 

23-h  9-E 

44 

16-d  7-C 

6-b  3-B 

45 

6-y  3-E 

15-1  7-A 

46 

17-x  7-D 

23-t  9-D 

47 

2-n  1-B 

15-0  7-C 

20  23 


SESSION  3 

SESSION  4 

Scene/Type 

Scene/Type 

10-t  4-D 

6-g  3-C 

15-d  7-C 

16-h  7-E 

20-u  8-D 

20-n  8-B 

23-a  9-B 

5-u  2-D 

*20-q  8-C 

14-e  6-B 

17-t  7-D 

2-j  1-A 

14-m  6-B 

16-x  7-D 

2-r  1-C 

20-h  8-E 

17-i  7-A 

15-f  7-A 

10-q  4-C 

14-v  6-D 

14-a  6-B 

10-h  4-E 

15-t  7-D 

23-d  9-C 

6-m  3-B 

14-f  6-A 

17-s  7-D 

20-O  8-C 

5-y  2-E 

17-j  7-A 

16-p  7-C 

#0-h  1-E 

23-n  9-B 

5-1  2-A 

10-g  4-C 

*23-s  9-D 

6-k  3-A 

16-e  7-B 

2-g  1-C 

10-f  4-A 

6-u  3-D 

6-w  3-B 

20-y  8-E 

14-1  6-A 

10-r  4-C 

20-r  8-C 

16-C  7-D 

10-n  4-B 

6-1  3-A 

6-j  3-A 

23-p  9-C 

23-q  9-C 

#0-S  1-D 

10-1  4-A 

10-0  4-C 

15-y  7-E 

6-x  3-D 

23-w  9-B 

2-y  1-E 

16-q  7-C 

16-u  7-D 

10-x  4-D 

6-r  3-C 

17-r  7-C 

23-j  9-A 

2-f  1-A 

20-c  8-D 

5-c  2-D 

5-d  2-C 

2-in  1-B 

2-C  1-D 

*23-s  9-D 

17-1  7-A 

5-n  2-B 

*20-q  8-C 

16-y  7-E 

5-a  2-B 

2-w  1-B 

20-1  8-A 

16-s  7-D 

14-t  6-D 

23-f  9-A 

6-i  3-E 

6-s  3-D 

5-k  2-A 

14-p  6-C 

15-q  7-C 

5-m  2-B 

14-c  6-D 

15-s  7-D 

6-0  3-C 

14-0  6-C 

2-1  1-A 

17-m  7-B 
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48 

20-p 

8-C 

20-b 

8-B 

15-n 

7-B 

14-k 

6-A 

49 

16-b 

7-B 

17-c 

7-D 

20-j 

8 -A 

20-t 

8-D 

50 

2-u 

1-D 

14-w 

6-B 

16-t 

7-D 

17-a 

7-B 

51 

5-h 

2-E 

23-0 

9-C 

6-a 

3-B 

23-r 

9-C 

52 

23-b 

9-B 

10-j 

4 -A 

5-t 

2-D 

10-y 

4-E 

53 

17-d 

7-C 

16-w 

7-B 

17-w 

7-B 

5-j 

2-A 

54 

6-f 

3-A 

2-d 

1-C 

10-p 

4-C 

15-w 

7-B 

55 

16-v 

7-D 

15-i 

7-E 

5-e 

2-B 

10-s 

4-D 

56 

14-q 

6-C 

5-g 

2-C 

6-d 

3-C 

15-j 

7-A 

57 

23-m 

9-B 

16-i 

7-E 

16-f 

7-A 

2-q 

1-C 

58 

*20-d 

8-C 

2-x 

1-D 

10-m 

4-B 

15-b 

7-B 

59 

2-a 

1-B 

16-0 

7-C 

2-t 

1-D 

23-c 

9-D 

60 

5-r 

2-C 

10-k 

4 -A 

6-n 

3-B 

17-b 

7-B 

61 

16-1 

7-A 

15-u 

7-D 

2-s 

1-D 

10-u 

4-D 

62 

#0-t 

1-D 

5-b 

2-B 

14-b 

6-B 

5-w 

2-B 

63 

17-p 

7-C 

15-r 

7-C 

17-v 

7-D 

17-g 

7-C 

64 

20-x 

8-D 

*23-u 

9-D 

2-0 

1-C 

20-w 

8-B 

65 

15-m 

7-B 

15-g 

7-C 
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PRESENTATION  ORDER  —  ORANGE  TEAM 
HRCs  3  4  9  11  12  17  18  20  21  25 


SESSION  1 

SESSION  2 

SESSION  3 

Scene/Type 

Scene/Type 

Scene/Type 

1 

4-v  2-D 

20-t  8-D 

*21-0 

8-C 

2 

20-i  8-E 

4-g  2-C 

3-i 

1-E 

3 

3-s  1-D 

25-n  9-B 

25-c 

9-D 

4 

4-y  2-E 

17-v  7-D 

9-w 

4-B 

5 

9-n  4-B 

3-0  1-C 

17-j 

7-A 

6 

20-y  8-E 

4-s  2-D 

4-d 

2-C 

7 

17-x  7-D 

18-w  7-B 

21-s 

8-D 

8 

12-f  5-A 

25-p  9-C 

9-1 

4-A 

9 

9-q  4-C 

20-e  8-B 

25-d 

9-C 

10 

20-h  8-E 

4-q  2-C 

11-1 

5-A 

11 

25-in  9-B 

3-k  1-A 

21-a 

8-B 

12 

#0-h  1-E 

9-h  4-E 

11-u 

5-D 

13 

18-k  7-A 

12-x  5-D 

4-k 

2-A 

14 

11-m  5-B 

20-d  8-C 

11-t 

5-D 

15 

18-p  7-C 

3-u  1-D 

3-r 

1-C 

16 

3-t  1-D 

9-b  4-B 

9-C 

4-D 

17 

21-g  8-C 

25-y  9-E 

4-6 

2-C 

18 

3-h  1-E 

3-b  1-B 

20-w 

8-B 

19 

9-0  4-C 

17-r  7-C 

25-u 

9-D 

20 

21-w  8-B 

25-x  9-D 

18-y 

7-E 

21 

17-f  7-A 

3-g  1-C 

25-g 

9-C 

22 

3-y  1-E 

17-e  7-B 

20-u 

8-D 

23 

9-a  4-B 

12-i  5-E 

12-e 

5-B 

24 

11-f  5-A 

25-r  9-C 

20-k 

8-E 

25 

4-C  2-D 

20-1  8-A 

11-g 

5-C 

26 

17-y  7-E 

9-r  4-C 

21-f 

8-A 

27 

11-n  5-B 

18-x  7-D 

4-a 

2-B 

28 

25-f  9-A 

9-m  4-B 

25-t 

9-D 

29 

20-g  8-C 

18-h  7-E 

4-b 

2-B 

30 

11-c  5-D 

21-d  8-C 

21-u 

8-D 

31 

25-1  9-A 

25-i  9-E 

12-n 

5-B 

32 

12-y  5-E 

*21-p  8-C 

4-i 

2-E 

33 

25-j  9-A 

17-t  7-D 

*21-0 

8-C 

34 

3-x  1-D 

25-0  9-C 

18-u 

7-D 

35 

18-0  7-C 

17-w  7-B 

9-d 

4-C 

36 

21-m  8-B 

21-y  8-E 

11-y 

5-E 

37 

12-1  5-A 

25-e  9-B 

3-p 

1-C 

38 

20-r  8-C 

21-q  8-C 

12-t 

5-D 

39 

4-n  2-B 

ll-w  5-B 

21-i 

8-E 

40 

*25-v  9-D 

9-f  4-A 

18-s 

7-D 

41 

11-k  5-A 

11-a  5-B 

12-in 

5-B 

42 

9-u  4-D 

21-t  8-D 

20-j 

8-A 

43 

12-j  5-A 

9-e  4-B 

#0-c 

1-D 

44 

17-m  7-B 

12-s  5-D 

18-b 

7-B 

45 

4-x  2-D 

20-m  8-B 

21-r 

8-C 

46 

18-1  7-A 

17-p  7-C 

9-k 

4-A 

47 

11-d  5-C 

9-x  4-D 

17-a 

7-B 

TAPES 


SESSION  4 
Scene/Type 

21-b  8-B 

11- r  5-C 
21-k  8-A 

#0-t  1-D 

17- d  7-C 

9-s  4-D 

4-j  2-A 

12- 0  5-C 

18- v  7-D 

11- g  5-C 

20-v  8-D 
3-e  1-B 

12- q  5-C 

3-m  1-B 

12-p  5-C 

3- a  1-B 
18-g  7-C 

4- 1  2-A 

9-i  4-E 

20-O  8-C 
9-j  4-A 

18-e  7-B 

*20-p  8-C 

17- n  7-B 

3- c  1-D 

11- p  5-C 

20- x  8-D 

12- a  5-B 

18- f  7-A 
25-h  9-E 

4- e  2-B 

9-p  4-C 

4-f  2-A 

9-y  4-E 

11-b  5-B 
4-p  2-C 

18-t  7-D 

11- j  5-A 
18-a  7-B 

21- c  8-D 

4-r  2-C 

17-h  7-E 
20-a  8-B 
17-k  7-A 

12- b  5-B 
20-c  8-D 
25-q  9-C 
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48 

17-1 

7 -A 

21-n 

8-B 

49 

3-v 

1-D 

12-g 

5-C 

50 

20-q 

8-C 

25-w 

9-B 

51 

9-t 

4-D 

18-d 

7-C 

52 

3-d 

1-C 

4-w 

2-B 

53 

12-v 

5-D 

25-s 

9-D 

54 

3-q 

1-C 

11-e 

5-B 

55 

21-x 

8-D 

4-h 

2-E 

56 

11-i 

5-E 

17-b 

7-B 

57 

17-s 

7-D 

11-v 

5-D 

58 

21-1 

8 -A 

17-g 

7-C 

59 

18-c 

7-D 

12-u 

5-D 

60 

12 -w 

5-B 

17-i 

7-E 

61 

17-0 

7-C 

★  21-p 

8-C 

62 

*25-v 

9-D 

12-h 

5-E 

63 

18-n 

7-B 

25-a 

9-B 

64 

3-f 

1-A 

#0-s 

1-D 

65 

4-m 

2-B 

12-k 

5 -A 

18-i 

7-E 

20-n 

8-B 

11-0 

5-C 

25-k 

9 -A 

20-f 

8 -A 

21-v 

8-D 

11-h 

5-E 

9-g 

4-C 

21-j 

8 -A 

17-c 

7-D 

3-n 

1-B 

21-h 

8-E 

4-u 

2-D 

3-1 

1-A 

18-m 

7-B 

11-s 

5-D 

12-d 

5-C 

18-r 

7-C 

20-s 

8-D 

3-w 

1-B 

25-b 

9-B 

18-q 

7-C 

12-r 

5-C 

21-e 

8-B 

9-v 

4-D 

12-c 

5-D 

17-q 

7-C 

18-j 

7 -A 

4-t 

2-D 

11-x 

5-D 

*20-p 

8-C 

20-b 

8-B 

3-j 

1-A 

17-u 

7-D 
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APPENDIXG  -  EQUIVALENT  METHOD  of  VIEWER  RANDOMIZATION 

This  appendix  presents  a  method  of  viewer  randomization  that  is  equivalent  to  that 
proposed  in  Section  2.9.2  of  the  Subjective  Test  Plan.  The  new  method  permits  gradual 
viewer  recruiting  through  sequential  assignment  to  random  teams  and  viewing  positions. 
This  method  was  discussed  and  accepted  by  the  full  Working  Group  at  the  November  10, 
1993  meeting  in  San  Jose,  CA. 

At  the  time  this  appendix  was  prepared,  it  was  known  that  each  lab  would  have  all  12 
subjective  viewing  tapes  at  the  same  time.  Therefore,  it  is  possible  to  randomize  the  tape 
teams  (RED,  GREEN,  and  ORANGE)  and  their  respective  random  orderings  (e.g.  1423 
2134  3214  etc.)  and  assign  a  particular  sub-team  (e.g.  G1  G4  G2  G3)  to  a  viewer  as  he  is 
recruited.  This  preserves  the  balance  achieved  through  randomization  and  allows  the  labs 
to  go  forward  with  the  testing  while  the  recruiting  process  continues. 

The  same  random  numbers  that  appear  in  Table  7  of  Section  2.9.2  will  be  used.  They  will 
now  be  used  to  order  the  sub-teams,  however,  instead  of  the  viewers.  The  viewers  will 
receive  numbers  from  the  Table  7  as  they  are  recruited  (or  scheduled). 

Each  viewer  will  be  given  a  number  from  Table  7  assigning  each  to  a  sub-team  and  seat 
position.  The  assignment  will  be  made  according  to  Table  7  (for  each  lab)  proceeding 
down  the  columns  and  disregarding  any  numbers  beyond  36  (or  beyond  the  expected 
number  of  viewers).  For  example,  the  first-recruited  subject  for  Lab  X  will  be  viewer  #8 
and  the  second-recruited  subject  for  Lab  X  will  be  viewer  #3. 

Each  sub-team  is  composed  of  three  viewers  and  hence  will  have  three  numbers 
associated  with  it  (one  for  the  viewer  assigned  to  the  Left  seat,  one  for  the  Center  seat, 
and  one  for  the  Right  seat).  Within  each  sub-team  of  three  subjects  the  first-listed  subject 
will  sit  in  the  Left  chair  in  each  session,  the  second-listed  subject  will  sit  in  the  Center 
chair,  and  the  third-listed  subject  will  sit  in  the  Right  chair.  For  example: 

viewer  Numberings  for  LabX  Viewer  Numberings  for  LabY 


Viewer 

Number  Subteam 

L  C  R  (tape  order) 

1,2,3  G1G4G2G3 

4,5,6  G2G1G3G4 

7,8,9  G3G2G1G4 


Viewer 

Number 

Subceam 

L  C  R 

(tape  order) 

1,2,3 

03010204 

4,5,6 

04010203 

7,8,9 
Etc . 

01020304 

34,35,36  R3R4R1R2 

Accordingly,  the  first-recruited  subject  for  Lab  X  (Viewer  #8)  will  be  assigned  the  sub¬ 
team  G3G2G1G4  and  will  sit  in  the  Center  seat.  Similarly,  the  second-recruited  subject 
(Viewer  #3)  will  be  assigned  to  sub-team  G1G4G2G3  and  will  sit  in  the  Right  seat. 

In  order  to  facilitate  scheduling  subjects,  substitutions  may  be  allowed  within  sub-teams 
of  the  same  "color"  (R,G,0),  but  not  across  "colors."  For  example,  if  the  viewer  assigned 
to  sub-team/seat  number  2  (i.e.  viewer  #2),  and  scheduled  to  view  tapes  G1G4G2G3,  was 
not  able  to  make  the  scheduled  showing,  the  lab  could  fill  the  vacant  slot  with  any  other 
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viewer  who  was  scheduled  to  view  GREEN  tapes.  However,  no  more  than  three  such 
substitutions  will  be  allowed  within  any  color  group. 

This  simpler  and  potentially  more  convenient  way  to  randomize  viewers  is  expected  to 
facilitate  completion  of  the  testing  in  a  timely  manner. 
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APPENDDC  H  -  CRITERIA  AND  THRESHOLDS  FOR  NORMAL  VISION 

This  appendix  gives  the  criteria  used  to  determine  whether  or  not  each  candidate  viewer 
possesses  normal  vision.  The  test  plan  requires  use  of  scores  from  only  those  viewers 
determined  to  have  normal  visual  acuity  and  color  discrimination.  The  scores  of  viewers 
who  fail  either  the  acuity  or  color  tests  are  set  aside  from  the  main  analysis. 

When  attempting  to  determine  the  normal  threshold  scores  on  the  acuity  tests,  it  was 
found  that  a  consensus  did  not  exist  among  the  optometrists  consulted.  Furthermore,  the 
composition  of  color  vision  test  materials  vary  by  supplier  and  the  normal  vision 
thresholds  vary  by  specific  test.  This  appendix  documents  the  thresholds  (and  tests)  that 
were  used  in  association  with  this  test  plan. 

The  table  below  defines  normal  vision  for  the  purpose  of  this  test  program. 


CRITERIA 

LAB 

THRESHOLD  SCORES 

Acuity  at  6H 
(seated) 

ITS 

zero  errors  on  line  7 
(20/25  or  better) 

Acuity  at  6H 
(seated) 

GTE 

zero  errors  on  line  7 
(20/25  or  better) 

Acuity  at  6H 
seated 
standing 

DIS/NCS 

zero  errors  on  line  7 
<1  error  on  line  7 

Color  Discrimination 

Pseudo-Isochromatic  Plates 
for  Testing  Color  Perception 

ITS 

>5  correct  out  of  7 
(sub-set  of  plates:  86, 

15,  56,  57,56, 47,  74) 

Pseudo-Isochromatic  Plates 
for  Testing  Color  Perception 

GTE 

>15  correct  out  of  18 
(complete  set  of  plates) 

Pseudo-Isochromatic  Plates 
for  Testing  Color  Perception 

DIS/NCS 

>10  correct  out  of  14 
(complete  set  of  plates) 
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1.  Introduction 


The  Institute  for  Telecommunication  Sciences  has  completed  subjective  testing  of  the 
T1A1.5  video.  As  required  by  the  Subjective  Test  Plan  (TlAl  .5/93-014  R5),  both  the  raw 
subjective  test  scores  and  summary  data  have  been  made  available  to  T1A1.5 
participants.  This  document  describes  how  to  access  the  data  and  provides  a  summary 

of  the  data. 


2.  Data  Access 


The  data  has  been  posted  on  a  Unix  workstation  at  ITS.  It  can  be  accessed  in  the 
following  manner  from  any  computer  with  Internet  access: 

ftp  ntia.its.bldrdocgov 
Name:  anonymous 
Password:  <Yoxu:  email  address> 


ftp>  cd  dist/tlal/subjective 
ftp>  ascii 

ftp>  get  <filename> 


ftp>  bye 

It  is  suggested  that  the  files  be  transferred  using  ascii  mode  to  prevent  transmission 
errors.  The  read.me  file  contains  a  complete  description  of  all  the  subjective  data  files 
available  for  transfer.  A  description  of  each  file  is  also  included  as  part  of  this  document. 


The  following  files  are  available  in  this  directory: 


DEMOG 

QUALIFY 

REPEAT 

SCORES 

SUMMARYl 

SUMMARY2 

read.me 

scenes 


Demographics  Information  for  all  viewers 

Viewers  scores,  NULL  checks  &  repeat  checks  excluded. 

Viewers  scores,  repeat  checks  only. 

All  viewers  scores,  including  order  viewed  information 
Summary  of  all  HRCs,  scenes  'a'  through 'm' 

Summary  of  all  HRCs,  scenes  'n'  through  y 

''Read  me"  information  file 

Scene  name  to  letter  cross  reference 


HRCOl 

HRC.02 

HRC.03 

HRC.04 

HRC.05 

HRC.06 

HRC.07 

HRC.08 

HRC.09 

HRCIO 

HRCll 


HRC 1  summary  table:  viewers'  scores  +  summary  statistics 
HRC  2  summary  table:  viewers'  scores  +  summary  statistics 
HRC  3  summary  table:  viewers'  scores  +  siunmary  statistics 
HRC  4  summary  table:  viewers'  scores  +  summary  statistics 
HRC  5  summary  table:  viewers'  scores  +  summary  statistics 
HRC  6  summary  table:  viewers'  scores  +  summary  statistics 
HRC  7  summary  table:  viewers'  scores  +  summary  statistics 
HRC  8  summary  table:  viewers'  scores  +  summairy  statistics 
HRC  9  summary  table:  viewers'  scores  +  summary  statistics 
HRC  10  summary  table:  viewers'  scores  +  summary  statistics 
HRC  11  summary  table:  viewers'  scores  +  summary  statistics 
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HRC.12  HRC 12  summary  table: 

HRC13  HRC  13  summary  table: 

HRC.14  HRC  14  summary  table: 

HRC.  15  HRC  15  summary  table: 

HRC.16  HRC  16  summary  table: 

HRC.17  HRC  17  summary  table: 

HRC.18  HRC  18  summary  table: 

HRC.19  HRC  19  summary  table: 

HRC.20  HRC  20  summary  table: 

HRC.21  HRC  21  summary  table: 

HRC.22  HRC  22  summary  table: 

HRC.23  HRC  23  summary  table: 

HRC.24  HRC  24  summary  table: 

HRC.25  HRC  25  summary  table: 


viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 
viewers'  scores  +  summary  statistics 


Please  direct  questions  or  problems  to: 

Margaret  Pinson  303-497-3579  margaret@its.bldrdoc.gov 

Coleen  Jones  303-497-3764  qones@its.bldrdoc.gov 


3.  Data  Summary 


3.1  Read.meFile 

A  "READ .ME"  fQe  is  included  which  briefly  describes  the  contents  of  each  file.  This 
readme  file  is  similar  in  content  to  this  paper. 

3.2  Subjective  Test  Results  Summary 

Files  'SUMMARYl'  and  'SUMMARY2'  display  the  mean,  standard  deviation, 
standard  error,  minimum,  and  maximum  score  values  for  each  scene/HRC  combination. 
The  ten  qualifying  viewers'  scores  were  used  for  these  calculations.  These  statistics  are 
displayed  to  three  decimal  places  in  the  table.  Information  is  split  into  two  files.  The  first 
file,  'SUMMARYl',  contains  summary  information  for  scenes  A  through  M;  the  second 
file,  'SUMMARY2',  contains  summary  information  for  scenes  N  through  Y. 

These  files  are  listed  below  on  fadng  pages,  for  your  convenience. 
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SUMMARYl 


HRC  abcdefghij  kl  m 


1  MEAN  4.90  5.00  5.00  5.00  5.00  5.00  5.00  5.00  4.90  5.00  5.00  5.00  5.00 

S.D.  0.32  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.32  0.00  0.00  0.00  0.00 

S.E.  0.10  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.10  0.00  0.00  0.00  0.00 

MAX  5555555555555 
min  4555555545555 


HRC  abcdefghij  kl  m 


2  MEAN  4.30  4.10  3.50  4.30  4.60  4.20  4.30  4.20  4.30  4.20  4.50  4.60  5.00 

S.D.  0.82  0.74  0.85  0.82  0.52  0.79  0.67  0.63  0.48  0.63  0.53  0.52  0.00 

S.E.  0.26  0.23  0.27  0.26  0.16  0.25  0.21  0.20  0.15  0.20  0.17  0.16  0.00 

MAX  5  555555555555 

MIN  3323433343445 


HRC  abcdefghij  kl  m 


3  MEAN  4.90  5.00  4.80  5.00  4.90  5.00  5.00  4.80  4.80  5.00  5.00  5.00  5.00 

S.D.  0.32  0.00  0.42  0.00  0.32  0.00  0.00  0.42  0.42  0.00  0.00  0.00  0.00 

S.E.  0.10  0.00  0.13  0.00  0.10  0.00  0.00  0.13  0.13  0.00  0.00  0.00  0.00 

MAX  5555555555555 
MIN  4545455445555 


HRC  abcdefghij  kl  m 


4  MEAN  3.05  2.60  2.65  2.60  2.10  3.05  2.65  1.90  1.25  3.10  3.50  3.60  1.90 

S.D.  0.89  0.94  0.75  0.99  0.85  0.76  0.81  0.72  0.44  0.64  0.61  0.75  0.85 

S.E.  0.20  0.21  0.17  0.22  0.19  0.17  0.18  0.16  0.10  0.14  0.14  0.17  0.19 

MAX  4444444324454 
MIN  1111111112221 


HRC  abcdefghij  kl  m 


5  MEAN  3.50  3.10  2.80  3.10  3.00  3.60  3.10  2.60  1.50  3.40  3.80  4.00  2.90 

S.D.  0.85  0.74  0.79  0.88  0.82  0.70  0.88  0.70  0.85  0.52  0.63  0.47  0.88 

S.E.  0.27  0.23  0.25  0.28  0.26  0.22  0.28  0.22  0.27  0.16  0.20  0.15  0.28 

MAX  54444543345  5  4 

MIN  2211232113331 
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SUMMARY2 


HRC  nopqr  st  uvwxy 


1  MEAN  5.00  4.80  5.00  4.90  5.00  4.40  5.00  5.00  4.90  5.00  5.00  4.90 

SD.  0.00  0.42  0.00  0.32  0.00  0.97  0.00  0.00  0.32  0.00  0.00  0.32 

S.E.  0.00  0.13  0.00  0.10  0.00  0.31  0.00  0.00  0.10  0.00  0.00  0.10 

MAX  555555555555 
MIN  545452554554 


HRC  nopqr  St  uvwxy 


2  MEAN  4.70  4.90  4.30  4.60  4.50  4.20  4.60  4.20  4.00  4.10  4.20  4.60 

S.D.  0.48  0.32  0.67  0.97  0.71  0.63  0.52  0.63  0.94  0.74  0.63  0.52 

S.E.  0.15  0.10  0.21  0.31  0.22  0.20  0.16  0.20  0.30  0.23  0.20  0.16 

MAX  5  55555555555 

min  443233432334 


HRC  nopqrst  uvwxy 

3  MEAN  5.00  4.80  4.60  4.90  5.00  4.80  5.00  4.80  4.60  4.80  5.00  4.50 

S.D.  0.00  0.42  0.70  0.32  0.00  0.42  0.00  0.42  0.97  0.42  0.00  0.53 

S.E.  0.00  0.13  0.22  0.10  0.00  0.13  0.00  0.13  0.31  0.13  0.00  0.17 

MAX  555555555555 
MIN  543454542454 

HRC  nopqrst  uvwxy 


4  MEAN  1.60  2.75  2.30  1.30  1.70  1.30  1.45  3.50  2.20  2.70  2.80  1.50 

S.D.  0.75  0.91  0.80  0.73  0.80  0.57  0.60  0.61  0.89  0.80  0.89  0.76 

S.E.  0.17  0.20  0.18  0.16  0.18  0.13  0.14  0.14  0.20  0.18  0.20  0.17 

MAX  344333344443 
MIN  111111121111 


HRC  nopqrstuvwxy 


5  MEAN  2.20  3.20  3.50  2.30  2.60  1.60  1.80  3.70  3.50  2.60  3.00  2.70 

S.D.  0.63  0.63  0.53  0.67  0.97  0.70  0.79  0.48  0.71  0.84  0.67  0.67 

S.E.  0.20  0.20  0.17  0.21  0.31  0.22  0.25  0.15  0.22  0.27  0.21  0.21 

MAX  344343345444 
MIN  123111133122 


SUMMARYl 


HRC  abcdcfghij  kl  m 


6  MEAN  3.10  2.30  2.30  2.50  1.90  3.40  2.10  2.00  1.20  2.80  3.00  3.40  1.50 

S.D.  0.57  0.48  0.82  0.85  0.74  0.70  0.57  0.82  0.42  0.63  0.47  0.70  0.71 

S.E.  0.18  0.15  0.26  0.27  0.23  0.22  0.18  0.26  0.13  0.20  0.15  0.22  0.22 

MAX  4334353324443 
min  2211131112221 


HRC  abcdef  ghij  kl  m 


7  MEAN  4.10  2.80  2.80  3.30  3.40  4.10  3.60  3.10  1.90  4.00  4.10  4.20  2.70 

S.D.  0.57  0.63  0.63  0.67  0.84  0.32  0.52  0.57  0.88  0.67  0.74  0.63  0.67 

S.E.  0.18  0.20  0.20  0.21  0.27  0.10  0.16  0.18  0.28  0.21  0.23  0.20  0.21 

MAX  5444454435  554 

MIN  3222243213332 


HRC  abcdef 


g 


h 


i 


k  1  m 


8  MEAN  4.10  3.70  3.30  3.40  3.40  4.00  3.90  2.80  1.90  4.20  4.20  4.30  3.30 

S.D.  0.74  0.48  0.82  0.70  0.84  0.47  0.57  0.63  0.99  0.63  0.42  0.48  0.48 

S.E.  0.23  0.15  0.26  0.22  0.27  0.15  0.18  0.20  0.31  0.20  0.13  0.15  0.15 

MAX  5444555445554 
MIN  3322233213443 


HRC  abcdefghij  kl  m 


9  MEAN  4.40  3.20  2.20  3.80  2.90  4.60  4.40  2.80  2.00  3.90  4.50  4.30  3.00 

S.D.  0.70  0.92  0.63  0.42  0.88  0.52  0.52  0.92  0.94  0.57  0.71  0.67  1.15 

S.E.  0.22  0.29  0.20  0.13  0.28  0.16  0.16  0.29  0.30  0.18  0.22  0.21  0.37 

max  5534455435554 
MIN  3213144113331 


HRC  abcdefghij  klm 


10  MEAN  4.90  3.80  3.10  3.40  3.80  4.40  4.20  3.20  2.40  4.30  4.40  4.70  3.60 

S.D.  0.32  0.79  0.88  0.52  0.79  0.70  0.63  0.63  0.97  0.48  0.52  0.48  0.70 

S.E.  0.10  0.25  0.28  0.16  0.25  0.22  0.20  0.20  0.31  0.15  0.16  0.15  0.22 

MAX  5544555445555 
min  4323333214443 
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SUMMARY2 


HRC  nopqrst  uvwxy 


6  MEAN  1.50  1.90  2.70  1.30  1.80  1.20  2.30  3.20  2.50  2.10  2.00  2.20 

S.D.  0.53  0.99  0.67  0.48  0.42  0.42  0.67  0.79  0.71  0.74  0.67  0.79 

S.E.  0.17  0.31  0.21  0.15  0.13  0.13  0.21  0.25  0.22  0.23  0.21  0.25 

MAX  244222344333 
MIN  112111122111 


HRC  nopqrst  uvwxy 


7  MEAN  2.90  3.30  3.40  2.50  2.90  2.00  2.70  4.20  3.10  3.10  3.60  3.10 

S.D.  0.57  0.67  0.70  0.53  0.88  0.67  0.82  0.42  0.88  0.74  0.52  0.99 

S.E.  0.18  0.21  0.22  0.17  0.28  0.21  0.26  0.13  0.28  0.23  0.16  0.31 

MAX  4443434  5  5445 

MIN  222211142232 

HRC  nopqrst  uvwxy 


8  MEAN  3.40  3.60  3.80  3.30  3.70  2.50  3.20  4.10  3.50  4.10  4.10  3.20 

S.D.  0.70  0.52  0.63  0.67  0.48  1.08  0.42  0.32  0.53  0.74  0.32  0.92 

S.E.  0.22  0.16  0.20  0.21  0.15  0.34  0.13  0.10  0.17  0.23  0.10  0.29 

MAX  445444454555 

MIN  233231343342 

HRC  nopqrst  uvwxy 

9  MEAN  3.00  3.20  3.20  2.40  3.30  2.10  2.80  4.40  3.10  3.30  3.60  3.40 

S.D.  0.47  0.63  0.63  1.07  0.67  0.88  0.63  0.70  0.99  0.67  0.70  1.07 

S.E.  0.15  0.20  0.20  0.34  0.21  0.28  0.20  0.22  0.31  0.21  0.22  0.34 

MAX  444443454455 

MIN  222121231231 

HRC  nopqrst  uvwxy 

10  MEAN  3.90  3.50  3.50  3.20  4.10  3.00  3.00  4.70  4.40  4.30  3.90  3.70 

S.D.  0.57  0.71  0.85  0.42  0.74  0.47  0.47  0.48  0.52  0.48  0.57  0.67 

S.E.  0.18  0.22  0.27  0.13  0.23  0.15  0.15  0.15  0.16  0.15  0.18  0.21 

MAX  545454455555 

MIN  322332244433 
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SUMMARY  1 


HRC  abcdefghij 


11  MEAN  2.40  2.90  1.60  2.20  1.60  2.70  2.00  2.00  1.30  1.90  2.90  1.90  1.40 

S.D.  0.84  0.74  0.52  0.92  0.70  0.82  1.05  0.82  0.48  0.88  0.88  0.74  0.52 

S.E.  0.27  0.23  0.16  0.29  0.22  0.26  0.33  0.26  0.15  0.28  0.28  0.23  0.16 

MAX  4423344323432 
MIN  121111111111^ 


HRC  abcdef  ghij  kl  m 

12  MEAN  2.10  2.30  1.40  2.30  1.60  2.60  2.40  1.80  1.00  2.30  2.70  2.90  1.50 

S.D.  0.74  1.16  0.52  0.95  1.07  0.84  0.97  0.63  0.00  0.95  0.82  0.57  0.53 

S.E.  0.23  0.37  0.16  0.30  0.34  0.27  0.31  0.20  0.00  0.30  0.26  0.18  0.17 

MAX  3424444314442 
MIN  1111111111121 


HRC  abcdefghij  kl  m 


13  MEAN  2.10  1.60  1.30  1.90  1.50  2.00  1.60  2.00  1.50  2.40  2.70  3.00  1.80 

S.D.  0.88  0.70  0.48  0.57  0.85  0.67  0.70  0.94  0.85  0.52  0.48  0.67  0.79 

S.E.  0.28  0.22  0.15  0.18  0.27  0.21  0.22  0.30  0.27  0.16  0.15  0.21  0.25 

MAX  4323333333343 
MIN  1111111112221 

HRC  abcdefghij  kl  m 

14  MEAN  1.40  1.90  1.50  2.00  1.90  1.90  2.10  2.00  1.80  3.00  2.40  2.80  2.80 

S.D.  0.52  0.88  0.71  0.82  0.74  0.57  0.88  0.67  0.63  0.67  0.70  0.63  0.79 

S.E.  0.16  0.28  0.22  0.26  0.23  0.18  0.28  0.21  0.20  0.21  0.22  0.20  0.25 

MAX  2333333334344 
MIN  1111111112121 

HRC  abcdefghij  kl  m 


15  MEAN  2.60  2.25  1.75  1.68  1.35  2.90  1.85  1.80  1.30  2.15  2.75  1.80  1.35 

S.D.  0.75  0.64  0.64  0.58  0.59  0.45  0.75  0.70  0.57  0.67  0.55  0.70  0.59 

S.E.  0.17  0.14  0.14  0.13  0.13  0.10  0.17  0.16  0.13  0.15  0.12  0.16  0.13 

MAX  4333343333333 
MIN  1111121111111 
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SUMMARY2 


HRC  nopqrstuvwxy 


11  MEAN  1.40  2.20  2.10  1.10  1.40  1.30  1.20  1.90  1.20  1.80  1.80  1.40 

S.D.  0.52  0.79  0.74  0.32  0.70  0.48  0.42  0.99  0.42  0.92  0.63  0.70 

S.E.  0.16  0.25  0.23  0.10  0.22  0.15  0.13  0.31  0.13  0.29  0.20  0.22 

MAX  233232232333 
MIN  111111111111 


HRC  nopqrstuvwxy 


12  MEAN  1.30  2.20  3.20  1.50  1.10  1.30  1.10  1.80  1.00  2.10  2.50  1.30 

S.D.  0.48  0.63  0.63  0.85  0.32  0.48  0.32  0.63  0.00  0.74  1.08  0.67 

S.E.  0.15  0.20  0.20  0.27  0.10  0.15  0.10  0.20  0.00  0.23  0.34  0.21 

MAX  234322231343 
MIN  112111111111 


HRC  nopqrstuvwxy 


13  MEAN  2.20  2.00  2.00  1.20  1.50  1.60  1.70  1.80  1.30  1.80  2.10  1.90 

S.D.  0.92  0.82  0.94  0.42  0.71  0.70  0.67  0.63  0.48  0.79  0.99  0.74 

S.E.  0.29  0.26  0.30  0.13  0.22  0.22  0.21  0.20  0.15  0.25  0.31  0.23 

MAX  333233332343 
MIN  111111111111 


HRC  nopqrst  uvwxy 


14  MEAN  2.70  1.80  1.60  1.60  2.00  1.70  1.60  1.60  1.70  2.00  2.20  2.80 

S.D.  0.82  0.63  0.70  0.70  0.82  0.67  0.52  0.84  0.82  0.94  0.92  0.63 

S.E.  0.26  0.20  0.22  0.22  0.26  0.21  0.16  0.27  0.26  0.30  0.29  0.20 

MAX  433333233334 
MIN  111111111112 

HRC  nopqrstuvwxy 


15  MEAN  1.25  1.45  1.85  1.15  1.45  1.40  1.30  2.55  1.35  2.00  1.30  1.40 

S.D.  0.44  0.60  0.75  0.37  0.69  0.60  0.47  0.51  0.49  0.73  0.47  0.68 

S.E.  0.10  0.14  0.17  0.08  0.15  0.13  0.11  0.11  0.11  0.16  0.11  0.15 

MAX  233233232323 
MIN  111111121111 
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HRC  abcdefghij  kl 


16  MEAN  2.80  2.80  2.30  2.20  1.70  2.60  2.50  2.20  1.20  2.40  2.40  2.60  1.30 

S.D.  0.79  0.92  0.82  0.63  0.67  0.52  0.85  1.23  0.42  0.84  0.97  0.70  0.67 

S.E.  0.25  0,29  0.26  0.20  0.21  0.16  0.27  0.39  0.13  0.27  0.31  0.22  0.21 

max  4443334424433 
MIN  2211121111111 


HRC  abcdefghij  kl  m 


17  MEAN  3.25  2.90  2.25  2.40  1.70  3.35  2.60  2.20  1.50  2.30  3.15  3.00  1.55 

S.D.  0.72  0.85  0.97  0.82  0.73  0.59  0.82  0.83  0.76  0.92  0.67  0.73  0.51 

S.E.  0.16  0.19  0.22  0.18  0.16  0.13  0.18  0.19  0.17  0.21  0.15  0.16  0.11 

MAX  4444344444542 
MIN  2111121111221 


HRC  abcdefghij  kl 


18  MEAN  4.30  2.70  2.40  2.80  2.50  4.20  3.60  2.20  1.20  2.90  3.40  3.90  1.60 

S.D.  0.67  0.82  0.84  0.79  0.85  0.63  0.70  0.79  0.42  0.74  0.52  0.57  0.70 

S.E.  0.21  0.26  0.27  0.25  0.27  0.20  0.22  0.25  0.13  0.23  0.16  0.18  0.22 

MAX  5444455324453 
MIN  3211133112  3  31 


HRC  abcdefghij  kl  m 


19  MEAN  3.60  2.60  2.00  2.90  2.50  3.70  3.50  2.40  1.80  3.10  3.20  3.80  1.80 

S.D.  0.52  0.84  0.82  0.57  0.85  0.48  0.53  0.70  0.79  0.74  0.42  0.63  0.92 

S.E.  0.16  0.27  0.26  0.18  0.27  0.15  0.17  0.22  0.25  0.23  0.13  0.20  0.29 

MAX  4434444334453 
MIN  3112133112331 


HRC  abcdefghij  kl  m 


20  MEAN  4.40  3.73  2.97  3.47  3.43  4.40  3.87  2.67  1.87  3.70  4.10  4.03  2.63 

S.D.  0.62  0.87  0.67  0.63  0.90  0.50  0.57  1.03  0.82  0.53  0.61  0.49  0.81 

S.E.  0.11  0.16  0.12  0.11  0.16  0.09  0.10  0.19  0.15  0.10  0.11  0.09  0.15 

MAX  5545555445554 
MIN  3112143113331 
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SUMMARY2 


HRC  nopqrstuvwxy 


16  MEAN  1.20  1.20  1.60  1.20  2.10  1.50  1.50  2.30  2.20  1.90  1.40  1.20 

S.D.  0.42  0.42  0.70  0.42  1.20  0.71  0.53  0.82  0.92  0.74  0.52  0.42 

S.E.  0.13  0.13  0.22  0.13  0.38  0.22  0.17  0.26  0.29  0.23  0.16  0.13 

MAX  223243234322 
MIN  111111111111 


HRC  nopqr  s  t  uvwxy 


17  MEAN  1.80  2.15  2.60  1.60  1.85  1.90  2.20  3.00  1.85  2.60  2.45  1.55 

S.D.  0.77  0.88  0.68  0.75  0.88  0.72  0.52  0.73  0.81  0.75  0.69  0.60 

S.E.  0.17  0.20  0.15  0.17  0.20  0.16  0.12  0.16  0.18  0.17  0.15  0.14 

MAX  344343343433 
MIN  111111121211 


HRC  nopqrstuvwxy 


18  MEAN  1.50  2.30  3.30  1.70  1.60  1.30  1.50  3.40  1.10  2.90  2.80  1.40 

S.D.  0.71  0.95  0.48  0.82  0.52  0.48  0.71  0.70  0.32  0.99  0.92  0.70 

S.E.  0.22  0.30  0.15  0.26  0.16  0.15  0.22  0.22  0.10  0.31  0.29  0.22 

MAX  344322342443 
MIN  113111121111 


HRC  nopqrstuvwxy 


19  MEAN  2.30  2.50  3.10  1.20  2.40  1.90  2.60  3.90  2.30  2.50  3.20  2.30 

S.D.  0.67  0.71  0.74  0.42  0.70  0.88  0.52  0.32  0.67  0.71  0.42  0.67 

S.E.  0.21  0.22  0.23  0.13  0.22  0.28  0.16  0.10  0.21  0.22  0.13  0.21 

MAX  334243343344 
MIN  112121231132 


HRC  nopqrst  uvwxy 

20  MEAN  2.13  3.17  3.70  2.50  3.23  3.17  2.97  4.03  3.17  3.57  3.57  2.70 

S.D.  0.86  0.87  0.65  0.86  0.73  0.99  0.85  0.56  0.53  0.73  0.57  0.92 

S.E.  0.16  0.16  0.12  0.16  0.13  0.18  0.16  0.10  0.10  0.13  0.10  0.17 

MAX  445445454555 
MIN  112121232  231 
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SUMMARYl 


HRC  abcdefghijklm 

21  MEAN  3.40  2.90  3.10  2.80  2.70  2.80  3.10  3.30  1.70  3.00  3.20  3.30  2.40 

S.D.  0.70  0.74  0.74  0.79  0.82  0.92  0.74  0.95  0.82  1.15  0.63  0.82  0.84 

S.E.  0.22  0.23  0.23  0.25  0.26  0.29  0.23  0.30  0.26  0.37  0.20  0.26  0.27 

MAX  4444444434444 
MIN  2221212111221 


HRC  abcdefghij  kl  m 


22  MEAN  4.30  3.30  3.10  4.00  4.00  4.70  4.40  3.00  3.10  4.30  4.20  4.40  3.40 

S.D.  0.48  0.67  0.57  0.67  0.67  0.48  0.52  0.67  0.57  0.67  0.42  0.52  0.70 

S.E.  0.15  0.21  0.18  0.21  0.21  0.15  0.16  0.21  0.18  0.21  0.13  0.16  0.22 

MAX  5445555445554 
MIN  4223344223442 


HRC  abcdefghij  kl  m 

23  MEAN  4.00  3.90  3.30  3.60  3.40  3.90  4.00  2.20  1.70  3.80  4.20  4.10  2.70 

S.D.  0.67  0.32  0.82  0.70  0.70  0.32  0.47  0.92  0.67  0.63  0.42  0.57  1.06 

S.E.  0.21  0.10  0.26  0.22  0.22  0.10  0.15  0.29  0.21  0.20  0.13  0.18  0.33 

MAX  5444445335555 
MIN  3322233113431 

HRC  abcdef  ghi  j  kl  m 


24  MEAN  4.70  3.90  3.70  4.00  3.90  4.50  4.70  3.50  2.90  4.10  4.40  4.20  3.80 

S.D.  0.48  0.32  0.82  0.00  0.32  0.53  0.48  0.71  0.88  0.57  0.70  0.42  0.42 

S.E.  0.15  0.10  0.26  0.00  0.10  0.17  0.15  0.22  0.28  0.18  0.22  0.13  0.13 

MAX  5454455445554 
MIN  4324344213343 


HRC  abcdefghij  kl  m 

25  MEAN  4.00  3.80  3.60  3.70  4.00  4.00  4.10  2.90  2.80  4.10  4.30  4.10  3.50 

S.D.  0.67  0.79  0.52  0.48  0.67  0.00  0.57  0.88  0.79  0.32  0.48  0.32  0.53 

S.E.  0.21  0.25  0.16  0.15  0.21  0.00  0.18  0.28  0.25  0.10  0.15  0.10  0.17 

MAX  5544545445554 
MIN  3233343124443 
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SUMMARY2 


HRC  nopqr  s  t  uvwxy 

21  MEAN  2.10  1.90  2.80  1.80  2.30  2.20  2.40  3.60  2.20  3.50  3.40  2.40 

S.D.  0.99  0.88  0.63  0.79  0.67  0.92  0.84  0.84  0.92  0.53  0.70  1.26 

S.E.  0.31  0.28  0.20  0.25  0.21  0.29  0.27  0.27  0.29  0.17  0.22  0.40 

MAX  434334454444 
MIN  112111121321 


HRC  nopqr  s  t  uvwxy 

22  MEAN  3.20  3.50  3.90  3.50  3.80  1.70  3.10  4.50  4.20  3.70  3.60  4.10 

S.D.  0.42  0.71  0.32  0.53  0.63  0.95  0.74  0.53  0.79  0.67  0.70  0.88 

S.E.  0.13  0.22  0.10  0.17  0.20  0.30  0.23  0.17  0.25  0.21  0.22  0.28 

MAX  454454455  5  45 

MIN  333331243323 

HRC  nopqr  s  t  uvwxy 

23  MEAN  2.00  3.50  3.80  3.10  3.60  2.20  3.50  4.00  3.80  3.60  3.60  3.20 

S.D.  0.94  0.53  0.79  0.99  0.70  1.03  0.53  0.47  0.42  0.70  0.70  0.63 

S.E.  0.30  0.17  0.25  0.31  0.22  0.33  0.17  0.15  0.13  0.22  0.22  0.20 

MAX  445544454444 
MIN  133221333222 


HRC  nopqr  s  t  uvwxy 

24  MEAN  3.90  3.70  4.10  2.80  4.00  3.20  4.30  4.60  3.90  3.90  3.90  3.70 

S.D.  0.99  0.67  0.57  0.63  0.67  0.92  0.48  0.52  0.74  0.57  0.57  0.67 

S.E.  0.31  0.21  0.18  0.20  0.21  0.29  0.15  0.16  0.23  0.18  0.18  0.21 

MAX  555454555555 
MIN  233232443333 


HRC  nopqr  s  t  uvwxy 

.25  MEAN  3.50  3.20  3.90  3.70  4.00  3.00  3.50  4.20  4.10  4.00  4.40  4.30 

S.D.  0.71  0.92  0.57  0.82  0.67  1.15  0.71  0.42  0.99  0.47  0.52  0.67 

S.E.  0.22  0.29  0.18  0.26  0.21  0.37  0.22  0.13  0.31  0.15  0.16  0.21 

MAX  445555455555 
MIN  213231242343 


3.3  Sample  HRC  data 

The  HRC  tables  are  separated  into  25  files,  each  listing  the  scores  associated  with  one 
HRC.  These  files  are  named  'HRCOl",  "HRC.02",  "HRC.03", ...  '•HRC.25".  The  suffix 
number  is  the  HRC  described  in  this  file.  The  format  of  these  files  is  described  m  the 
subjective  test  plan.  Notice  that  the  summary  statistics  listed  at  Ae  bottom  of  Ae  file  ^e 
computed  on  the  10  qualifying  viewers.  Scene  /  HRC  combinations  which  a  viewer  did 
not  score  are  marked  with  a  zero  (0).  These  scene  /  HRC  paris  are  not  used  in  die 
computation  of  summary  statistics.  The  column  "VIEWER  QUALIFY"  marks  the 
vaHdity  status  of  each  viewer:  'O'  for  Invalid  viewers;  T  for  vahd  or  qualified  viewers; 
for  alternate  viewers  (valid  but  extraneous). 

The  contents  of  the  ITS  file  HRC.20  are  displayed  below. 

HRCNO.20 

-VIEWER —  - SCENE-— -  REPEAT 

TEAM  ID  QUALIFY  abcdef  ghi  j  kl  mnopqr  St  uvwxy  o 

RED  80.Z  1  4433354323442224123454433  4 

09.Z  1  5334344224543233343243432  3 

37. Z  1  5423253113453234232353332  3 

39. Z  1  3433443224443233334444442  3 

38. Z  1  5433454314441144344443443  4 

40. Z  1  4433343444443333334343433)  3 

41 z  1  4423344424443222222243433  3 

43. Z  1  4434444224442144243343442  4 

46. Z  0  4433455334545343344453434  2 

47. Z  1  5533354124343325343343344  3 

48. Z  1  5434443434543244234443332  3 

06.Z  *  5323344233433233232343333  3 

82.Z  *  5433354214543213344343323  2 

44. Z  *  5534455335553344444455444  4 

-VIEWER —  - SCENE - -  REPEAT 

TEAM  ID  QUALIFY  abc  de  f  ghi  j  kl  mnopqr  s  t  uvwxy  d  q 

GREEN  13.Z  1  4234455323442123135243241  1  4 

15.Z  1  3343344424443343234343442  3  4 

17. Z  1  5535554314553245244443443  5  2 

18. Z  0  5334454313443244244344442  4  2 

90.Z  1  4424344113333113233433333  3  2 

22. Z  1  5544555224554244444454544  4  4 

23. Z  1  5444554314542234343443442  3  4 

24. Z  1  4433344334443334443444443  4  4 

50.Z  1  4433354113442124223243232  2  3 

54.Z  1  5433344213342234234243333  3  3 
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55.Z  1  4433344413444333343343333  3  3 

57.Z  *  3333333224342233233343332  2  3 

20. Z  •  3423445333443334233233443  3  4 

19.Z  *  5443455224542334242454333  3  3 

21. Z  *  5233444224542124324342342  2  4 


-VIEWER - SCENE -  REPEAT 

TEAM  IDQUALIFY  abc  de f  ghi  j  kl  mnopqr  s  t  uv wxy  p 


ORANGE  25.Z 

1 

4234354425452343235253453 

5 

26.Z 

0 

2323222342432122232333312 

3 

27.Z 

0 

5424545235532423342442333 

4 

28.Z 

1 

4433344234442244243344442 

3 

29.Z 

-  1 

5444555324542244443444443 

4 

30.Z 

1 

5434344334543344433344433 

4 

31  .z 

1 

4313344213443123232243332 

3 

32.Z 

0 

5542255333444344251234443 

4 

33.Z 

0 

4322233112331113122133331 

3 

34.Z 

1 

5434443334444434334243443 

4 

62.Z 

1 

4434444424444244321232444 

4 

63.Z 

1 

43221. 53123331 144122332331 

4 

69.Z 

1 

51 43 3 4 3314432444241253242 

4 

71  .z 

1 

5434444313442234333233434 

4 

a  bcdefghijklm 
MEAN  4.40  3.73  2.97  3.47  3.43  4.40  3.87  2.67  1.87  3.70  4.10  4.03  2.63 

S.D.  0.62  0.87  0.67  0.63  0.90  0.50  0.57  1.03  0.82  0.53  0.61  0.49  0.81 

S.E.  0.11  0.16  0.12  0.11  0.16  0.09  0.10  0.19  0.15  0.10  0.11  0.09  0.15 

MAX  5545555  445554 

MIN  3112143113331 


MEAN 

S.D. 

S.E. 

MAX 

MIN 


nopqrstuvwxy 
2.13  3.17  3.70  2.50  3.23  3.17  2.97  4.03  3.17  3.57  3.57  2.70 

0.86  0.87  0.65  0.86  0.73  0.99  0.85  0.56  0.53  0.73  0.57  0.92 

0.16  0.16  0.12  0.16  0.13  0.18  0.16  0.10  0.10  0.13  0.10  0.17 

445445454555 
112121232231 
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3.4  Demographic  Data 

File  T)EMOG'  contains  demographics  for  each  viewer.  This  file  format  is  not 
specified  in  the  subjective  test  plan.  'TD"  is  the  number  and  lab  code  associated  with 
each  viewer.  NTIA  /  ITS  is  lab  "z".  Note  that  viewer  numbers  were  not  assigned 
sequentially.  'TDecade"  is  listed  for  brevity;  the  questionnaires  listed  a  range  of  ages 
corresponding  to  that  decade.  'Industry"  and  'Job"  include  the  entire  text  listed  on  the 
questiormaire.  "Eye  Test"  is  #/#  where  the  first  number  is  the  last  line  of  the  eye  chart 
completely  read  and  the  second  number  is  the  number  of  color  plates  missed.  NTTA/ITS 
viewers  were  allowed  to  miss  2  of  8  color  plates. 

The  entire  contents  of  the  NTIA  /  ITS  demographics  file  are  listed  below. 

RED  TEAM 

PAST  EYE  SESSION 


ID 

DECADE  GENDER 

INDUSTRY 

JOB 

VIDEO 

TEST 

SEAT 

ORDER 

80.Z 

30 

female 

Government 

Tedmical 

Recent 

9/2 

center 

2431 

09.2 

50 

mcile 

Government 

Prof^sional 

No 

9/0 

right 

2431 

37j2 

50 

male 

Government 

Executive/ 

Managerial 

Yes 

9/2 

left 

3142 

39.Z 

40 

female 

Government 

Professioruil 

Yes 

8/2 

right 

3142 

38.Z 

40 

male 

Government 

Technical 

No 

9/1 

center 

3142 

40.2 

40 

female 

Government 

Administrative/  No 

9/1 

left 

2413 

White  Collar 

41je 

30 

male 

Government 

Technical 

Recent 

9/0 

center 

2413 

43.2 

40 

male 

Engineering/ 

Architecture 

Tedmical 

No 

10/0 

left 

2431 

46.2 

60 

male 

Government 

Laborer 

No 

7/2 

left 

4231 

47.2 

50 

male 

Electronics/ 

Computers 

Electronics/ 

Professional 

Recent 

9/1 

center 

4231 

48.2 

20 

female 

Technical 

Recent 

8/2 

right 

4231 

06 

30 

Computers 

male 

Telecommunication/  Technical 

•No 

8/0 

right 

2413 

Utilities 

82.2 

60 

male 

female 

Government 

Executive/ 

Managerial 

Recent 

9/1 

left 

4231 

44.2 

60 

Government 

Secretarial 

No 

7/1 

center 

2431 

GREEN  TEAM 

PAST 

EYE 

SESSION 

ID 

DECADE  GENDER 

INDUSTRY 

JOB 

VIDEO 

TEST 

SEAT 

ORDER 

13.2 

40 

male 

Government 

Technical 

No 

9/1 

left 

4321 

15.2 

30 

female 

Government 

Secretarial 

No 

8/1 

right 

4321 

17.2 

40 

female 

Personal/ 

Administrative/ 

No 

8/2 

right 

1243 
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I 


18j; 

50 

male 

Business  Services 
..  Electronics/ 

White  Collar 
Administrative/ 

Yes 

7/4 

right 

1243 

90i 

30 

male 

Computers 

Engineering/ 

White  Collar 
Professional 

Yes 

10/1 

right 

1243 

22je 

40 

male 

Ardiitecture 

Government 

Technical 

Recent 

8/1 

left 

2341 

23^ 

50 

male 

Government 

Executive/ 

Recent 

9/1 

center 

2341 

24^ 

50 

male 

Government 

Managerial 

Administrative/ 

No 

9/1 

right 

2341 

50^ 

60 

male 

Government 

White  Collar 
Executive/ 

Recent 

9/1 

center 

4321 

54^ 

60 

male 

Managerial 

Electronics/ 

Executive/ 

Recent 

7/1 

right 

1243 

55^ 

30 

male 

Computers 

Government 

Support 

Telecommunication 

Maiutgerial 

Qerical/ 

No 

9/2 

left 

2431 

57a, 

30 

male 

/  Technical 

Yes 

9/0 

right 

2431 

20a 

40 

male 

Utilities 

Electronics/ 

Prof^sional 

No 

8/0 

center 

2431 

19a 

40 

male 

v^uixipuicr^ 

Telecommimication/  Technical 

Recent 

7/1 

left 

2431 

21a 

70 

male 

Utilities 

Government 

aerical/ 

No 

7/2 

right 

2431 

ID 

DECADE  GENDER 

Support 

ORANGE  TEAM 

INDUSTRY  JOB 

PAST 

VIDEO 

EYE 

TEST 

SESSION 

SEAT  ORDER 

25  J5 

40 

male 

Engineering/ 

Administrative/ 

No 

10/0 

left 

1324 

26^ 

20 

female 

Architecture 

Government 

White  Collar 
Administrative/ 

Recent 

10/0 

center 

1324 

27a 

50 

female 

Government 

White  Collar 
Qerical/ 

Suppxjrt 

Technical 

No 

9/1 

right 

1324 

28a 

30 

male 

Electronics/ 

No 

10/1 

left 

4132 

29a 

50 

male 

Computers 

Government 

Qerical/ 

Support 

Technical 

Recent 

8/0 

center 

4132 

80a 

50 

male 

Government 

No 

9/1 

right 

4132 

31a 

30 

male 

Government 

Technical 

No 

8/0 

left 

4312 

32a 

50 

female 

Government 

Professional 

Recent 

8/2 

center 

4312 

33a 

30 

male 

Engineering/ 

Executive/ 

Recent 

6/1 

right 

4312 

3ia 

40 

female 

Ardiitecture 

Government 

Managerial 

Administrative/ 

No 

9/2 

left 

3241 

62a 

30 

female 

Government 

\^te  Collar 
Secretarial 

No 

9/0 

center 

1324 

63a 

20 

male 

Government 

Technical 

No 

10/0 

right 

1324 

69a 

50 

female 

Government 

Administrative/ 

No 

10/1 

right 

4312 

71a 

40 

male 

Military 

White  Collar 
Police/ 

Recent 

9/0 

center 

3241 

Military 
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3.5  Raw  Scores  File 

The  file  'SCORES'  contains  all  scores  for  all  viewers  listed  one  per  line,  in  a  format 
suitable  for  loading  into  S.A.S.  or  similar  statistical  packages.  This  file  contains  for  each 
viewer,  the  following  five  columns: 

<viewer>  <order>  <hro  <scene>  <rating> 

<viewer>  is  the  viewer's  number  and  lab  affiliation  letter.  <order>  is  the  order  in 
which  the  viewer  saw  the  258  or  260  scene  /  HRC  pairs.  <hrc>  is  the  number,  '0'  through 
'25',  of  the  current  HRC.  HRC  '0'  marks  NULL  checks.  <scene>  is  the  letter,  'a'  through 
'/  of  the  current  scene.  <rating>  is  the  rating  given  to  this  scene  /  hrc  pair  by  the  viewer. 
'5'  is  Imperceptible;  '4'  is  Perceptible  but  not  Annoying;  '3'  is  Slightly  Annoying;  *2'  is 
Annoying;  and  '1'  is  Very  Annoying.  A  zero  in  this  column  marks  unrated  scene  /  HRC 
pairs. 

File  SCORES  includes  all  scene  /  HRC  pairs  viewed.  Note  that  consistency  checks 
will  appeair  twice  for  each  viewer  and  that  NULL  checks  are  included.  Also  note  that  all 
viewers'  data  are  listed  in  this  file  regardless  of  whether  the  viewer  is  valid,  invalid,  or 
extra. 

3.6  Qualifying  Scores  File 

The  file  'QUALIFY'  contains  all  scores,  except  NULL  checks  and  repeat  quality 
checks,  for  all  viewers  listed  one  per  line,  in  a  format  suitable  for  loading  into  S.A.S.  or 
similar  statistical  packages.  This  file  contains  for  each  viewer,  the  following  four 
columns: 

<viewer>  <hrc>  <scene>  <rating> 

<viewer>  is  the  viewer's  number  and  lab  affiliation  letter.  <hrc>  is  the  number,  '1' 
through  '25',  of  the  current  HRC.  <scene>  is  the  letter,  'a'  through '/  of  the  current  scene. 
<rating>  is  the  rating  given  to  this  scene  /  hrc  pair  by  the  viewer.  '5'  is  Imperceptible;  '4' 
is  Perceptible  but  not  Annoying;  '3'  is  Slightly  Annoying;  '2'  is  Annoying;  and  T  is  Very 
Annoying.  A  zero  in  this  column  marks  unrated  scene  /  HRC  combinations. 

File  QUALIFY  includes  all  scene  /  HRC  pairs  viewed.  Also  note  that  all  viewers'  data 
are  listed  in  this  file  regardless  of  whether  the  viewer  is  valid,  invalid,  or  extra. 

3.7  Repeat  Check  Scores  File 

The  file  'REPEAT'  contains  repeat  quality  check  scores  for  all  viewers  listed  one  per 
line,  in  a  format  suitable  for  loading  into  S.A.S.  or  similar  statistical  packages.  This  file 
contains  for  each  viewer,  the  following  four  columns: 

<viewer>  <hrc>  <scene>  <rating>  <order> 

<viewer>  is  tiie  viewer's  number  and  lab  affiliation  letter.  <hrc>  is  ffie  number,  '1' 
through  '25',  of  the  current  HRC.  <scene>  is  the  letter,  'a'  through  y  of  the  current  scene. 
<rating>  is  the  rating  given  to  this  scene  /  hrc  pair  by  the  viewer.  '5'  is  Imperceptible;  '4' 
is  Perceptible  but  not  Aimoying;  '3'  is  Slightly  Aimoying;  '2'  is  Annoying-  and  '1'  is  Very 
Annoying.  A  zero  in  this  column  marks  xmrated  scene  /  HRC  combinations.  <order>  is 


-17- 


the  order  in  which  the  repeat  checks  were  viewed.  T  marks  the  first  viewed;  'T  marks 
the  second  viewed. 

File  REPEAT  includes  all  scene  /  HRC  repeat  check  pairs.  Also  note  that  all  viewers' 
data  are  listed  in  this  file  regardless  of  whefiier  the  viewer  is  valid,  invalid,  or  extra. 

3.8  Raw  Data  Directory 

Directory  "raw"  contains  the  ratings  and  demographic  data  for  each  viewer  in  an 
unformatted  state. 


